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PREFACE 


This  document  was  prepared  by  the  Institute  for  Defense  Analyses  (IDA)  under  the 
task  order,  Ballistic  Missile  Defense  Organization  (BMDO)  Software  Technology,  and  ful¬ 
fills  an  objective  of  the  task,  to  prepare  “a  final  guide  for  BMD  Software  Capability  Eval¬ 
uation  (SCE)  teams  based  on  available  BMD  experience.” 

This  document  was  reviewed  by  the  IDA  research  staff  members:  Dr.  David  J.  Car¬ 
ney,  Ms.  Audrey  A.  Hook,  Dr.  Richard  J.  Ivanetich,  Dr.  Reginald  N.  Meeson,  Mr.  Michael 
S.  Nash,  Dr.  Charles  R  Pfleeger,  Mr.  Clyde  G.  Roby,  and  Dr.  Richard  L.  Wexelblat.  Their 
contributions  are  gratefully  acknowledged. 
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SUMMARY 


Software  in  Department  of  Defense  (DoD)  programs  has  become  a  critical  element 
over  the  past  10  years,  and  the  DoD  has  increased  its  emphasis  on  the  need  to  evaluate  and 
monitor  software  contractors  and  subcontractors.  Consequently,  the  Software  Engineering 
Institute  (SEI)  and  the  MITRE  Corporation  were  tasked  by  the  Air  Force  to  develop  a  soft¬ 
ware  evaluation  methodology;  the  results  were  the  SEI  process  maturity  model,  identifica¬ 
tion  of  key  process  areas  in  its  data  element  set,  and  the  method  of  evaluating,  the  Software 
Capability  Evaluation  (SCE).  The  Ballistic  Missile  Defense  Organization  (BMDO)  is  using 
the  SCE  method  to  determine  whether  developers  have  good  software  practices  to  reduce 
cost  and  schedule  overruns.  This  document  presents  the  results  of  an  ongoing  study  to 
assess  and  recommend  procedures  for  improving  the  effectiveness  of  the  SCEs,  based  on 
lessons  learned  from  conducting  them  on  BMD  programs.  It  will  be  used  to  supplement 
SEI’s  training  material  and  reports. 

In  conducting  SCEs,  the  BMDO  is  encouraging  continuous  process  improvement 
among  software  development  contractors.  SCEs  have  been  scheduled  throughout  the  BMD 
life  cycle,  and  BMD  program  offices  will  use  SCEs  for  input  to  the  Source  Selection  Eval¬ 
uation  Board  (SSEB)  as  well  as  for  monitoring  existing  contracts. 

Institute  for  Defense  Analyses  (IDA)  staff  members  were  trained  at  SEI  in  conduct¬ 
ing  SCEs  and  participated  as  technical  advisors  for  the  SCEs  performed  for  the  National 
Test  Facility,  Brilliant  Pebbles,  and  Brilliant  Eyes  programs.  The  lessons  learned  were 
incorporated  in  this  paper  along  with  findings  and  recommended  procedures  and  tech¬ 
niques.  They  are  presented  in  the  time  frame  of  when  activities  occur  in  the  SCE  process: 
before,  during,  or  after  conducting  the  SCE.  Following  is  a  brief  overview  of  the  lessons 
learned. 

Activities  Prior  To  Conducting  Evaluations  (In  Order  of  Occurrence) 

Develop  inputs  to  the  Request  for  Proposal  (RFP).  The  offerors  must  be  made 
aware  of  the  SCE  requirement  in  the  RFP.  Text  is  given  within  this  report  that  can  be  used 
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to  insert  the  SCE  requirements  notice  within  the  REP  and  the  associated  Instructions  for 
Preparation  of  Proposals. 

Establish  evaluation  criteria.  The  Source  Selection  Authority  (SSA)  will  use  the 
results  of  an  SCE  during  source  selection  as  either  a  specific  criterion  or  a  general  consid¬ 
eration.  The  SSA  will  request  the  evaluation  criteria  support  either  a  color  rating  or  a 
numerical  rating,  depending  on  whether  procurement  is  for  the  Air  Force  or  the  Army. 
Examples  of  both  rating  systems  are  included  in  this  report. 

Identify  specific  program  needs.  Each  BMD  program  office  may  have  specialized 
software  needs.  Additional  key  process  areas  may  be  added  to  the  standard  SEI  process 
maturity  model.  The  standard  key  process  areas  are  process  improvement,  defect  preven¬ 
tion,  process  measurement  and  analysis,  quality  management,  process  focus,  process  defi¬ 
nition,  training,  peer  reviews,  standards  and  procedures,  project  management,  project 
planning,  configuration  management,  quality  assurance,  and  subcontractor  management. 

Select  the  evaluation  team.  Detailed  qualifications  of  team  members  are  identified. 
Qualifications  include  training  and  technical  or  managerial  experience  in  the  area  of  soft¬ 
ware  development  or  acquisition  support.  BMD  teams  should  not  consist  of  members  from 
a  single  Service  or  Government  organization;  representatives  from  the  Air  Force,  Army, 
National  Test  Facility,  and  Federally  Funded  Research  and  Development  Centers  should  be 
included.  Other  considerations  include  team  skills,  leadership,  and  lack  of  conflict  of  inter¬ 
est.  All  SCE  team  members  will  be  required  to  sign  a  Procurement  Integrity  Certificate 
(PIC)  for  source  selection  at  hand.  The  PIC  requires  disclosure  of  financial  interest  in  any 
of  the  RFP  offerors.  Finally,  additional  training  should  be  arranged  for  SCE  teams  who 
have  little  SCE  experience  or  who  have  not  worked  together  on  an  actual  evaluation. 

Conducting  Evaluations 

Select  projects  to  be  evaluated.  The  contractor  will  provide  information  on  seven 
to  nine  projects.  The  SCE  team  will  select  three  projects  to  perform  an  SCE  during  a  three- 
day  visit.  The  projects  selected  must  adequately  represent  the  contractor’s  proposed  role 
and  help  to  judge  the  risk  associated  with  awarding  the  proposed  project  to  the  contractor. 
Guidance  for  selecting  the  appropriate  projects  is  provided. 

Establish  site  visit  schedule.  A  detailed  schedule  identifying  the  activities  during 
the  three-day  visit  is  described.  It  includes  a  list  of  interview  candidates,  a  prioritized  list 
of  topics  to  explore  during  each  interview,  and  the  allotted  time  for  document  reviews  and 
for  finalizing  SCE  findings  and  results. 
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Submit  document  request  to  the  contractor.  The  SCE  team  will  submit  a  list  of 
documents  to  be  made  available  to  the  SCE  team  prior  to  the  site  visit  and  during  the  site 
visit.  Prior  to  the  site  visit,  the  contractor  will  furnish  the  Software  Development  Plan, 
project  profiles,  an  organization  chart,  and  a  completed  SEI  questionnaire  form.  This  doc¬ 
umentation  is  used  by  the  SCE  team  to  plan  an  interview  schedule,  identify  issues  and  ques¬ 
tions,  and  form  a  basis  for  the  findings.  During  the  site  visit,  additional  documentation  is 
requested  to  substantiate  findings.  A  detailed  list  of  documents  is  identified  for  each  key 
process  area. 

Develop  entrance  briefings.  The  SCE  team  and  the  contractor  will  provide  an 
entrance  briefing  to  identify  the  scope  of  the  SCE  on  the  contractor’s  software  process. 
Recommended  topics  for  the  presentations  are  listed. 

Establish  interview  approach.  Interviews  are  conducted  with  project  personnel, 
and  project  documentation  is  reviewed  to  verify  the  adequacy  of  the  contractor’s  software 
practices.  All  information  gathered  during  the  site  will  be  documented  to  support  SCE  find¬ 
ings.  Recommendations  are  included  for  conducting  interviews,  establishing  SCE  team 
member  roles,  and  documenting  interview  notes. 

Activities  After  Conducting  an  Evaluation 

Final  report.  The  SCE  team  will  prepare  a  report  of  its  findings  for  the  contractor, 
the  SSEB,  the  BMD  program  office,  and  the  BMDO.  The  report  will  be  marked  CONFI¬ 
DENTIAL  and  distribution  will  be  controlled.  Descriptions  of  the  report  format  and  content 
are  included  in  this  report. 

Contractor  feedback  of  evaluation  results.  The  winning  contractors  will  be 
briefed  after  the  contract  is  awarded;  at  this  time  the  contractors  can  provide  feedback.  The 
losing  contractors  will  receive  a  very  high-level  overview  of  the  SCE  results.  Guidance  is 
included  for  providing  feedback  to  the  contractor. 

Use  of  evaluation  results  in  source  selection.  The  report  will  contain  a  summary 
of  the  contractor’s  strengths,  weaknesses,  and  improvement  plans  for  each  key  process 
area.  Details  of  the  SCE  report  are  included  along  with  a  description  of  the  summary  which 
will  be  used  by  the  SSEB. 

Use  of  evaluation  results  for  contract  monitoring.  An  SCE  may  be  used  to  help 
monitor  a  contract.  The  program  office  can  compare  the  results  of  the  evaluation  with  the 
contractor’s  process  improvement  plans.  If  there  are  discrepancies,  the  program  office 


should  notify  the  contractor  who  should  produce  an  acceptable  plan  to  mitigate  the  risks 
and  to  reduce  the  principal  weaknesses  over  the  length  of  the  contract. 

Registry  of  evaluation  results.  The  results  of  an  SCE  should  be  stored  by  BMDO 
in  a  repository  for  possible  reuse  in  later  procurements.  This  can  reduce  time,  effort,  and 
expense.  Using  an  SCE  repository  could  also  shorten  the  procurement  cycle  by  reducing 
the  number  of  new  SCEs  that  must  be  done. 
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1.  INTRODUCTION 


1.1  Purpose 

This  document  provides  the  means  for  improving  the  effectiveness  of  the  Ballistic 
Missile  Defense  (BMD)  Software  Capability  Evaluation  (SCE)  teams  and  the  quality  of 
their  results.  The  originator  of  the  SCE  concept  and  process,  the  Software  Engineering 
Institute  (SEI),  provides  basic  information  for  performing  SCEs  in  the  SEI  training  course 
[SEl  1991a].  In  practice,  additional  information  and  procedures  are  necessary  to  help 
achieve  the  best  possible  results. 

The  SCE  is  a  valuable  tool  that  assists  in  ensuring  that  the  Federal  Government  gets 
a  timely  quality  product  for  its  software  investment.  SCEs  are  being  used  in  the  BMD  pro¬ 
gram  as  part  of  two  distinct  activities:  source  selection  and  contract  monitoring.  When  used 
in  source  selection,  SCE  results  will  figure  into  the  overall  scores  of  the  offerors.  When 
SCEs  are  used  for  contract  monitoring,  the  program  office  can  check  whether  the  contrac¬ 
tor’s  software  development  process  has  been  maintained  and,  if  required,  whether  improve¬ 
ment  has  occurred. 

This  paper  is  for  use  within  the  BMD  program  by  teams  already  trained  at  SEI  to 
conduct  SCEs.  It  is  intended  to  supplement  SETs  training  material  and  reports  with  other 
information  and  procedures  the  BMD  teams  have  learned  and  used  through  their  experienc¬ 
es  performing  SCEs. 

1.2  Background 

In  the  last  decade,  the  visibility  and  importance  of  software  in  Department  of 
Defense  programs  have  increased  the  need  to  improve  the  Federal  Government’s  ability  to 
evaluate  and  monitor  software  contractors.  Responding  to  a  request  by  the  United  States 
Air  Force,  SEI,  with  assistance  from  the  MITRE  Corporation,  developed  a  software  evalu¬ 
ation  methodology. 


The  SEI  methodology  provides  five  levels  of  process  maturity,  each  associated  with 
key  process  areas  (KPA).  Refer  to  Table  1  for  details  [SEI  1991b]. 

Table  1.  SEI  Process  Maturity  Model 


Level 

Characteristics 

KPAs 

5 

Optimizing 

Continuous  process  improve¬ 
ment 

Process  Improvement 

Defect  Prevention 

4 

Managed 

Product  quality  planning  and 
tracking  of  measured  software 
processes 

Process  measurement  and 
analysis 

Quality  management 

3 

Defined 

Development  process  defined 
and  institutionalized  to  pro¬ 
vide  product  quality  control 

Process  focus 

Process  definition 

Training 

Peer  reviews 

Standards  and  procedures 

2 

Repeatable 

Management  oversight  and 
tracking  of  project;  stable 
planning  and  product  base¬ 
lines 

Project  management 

Project  planning 

Configuration  management 
Quality  assurance 

Subcontractor  management 

1 

Initial 

Ad  hoc  (unpredictable  and 
chaotic) 

“People” 

BMD  evaluation  teams  attend  a  four-day  training  course  at  SEI  to  learn  how  to 
apply  the  basic  methods  for  conducting  an  SCE.  The  SEI  course  reviews  the  main  process 
maturity  concepts,  teamwork  skills,  interview  practices,  scoring  project  questionnaires,  and 
the  exit  briefing  containing  the  SCE  findings.  These  methods  are  taught  at  a  high  level  and 
are  documented  in  a  training  manual  [SEI  1991a]. 

The  Institute  for  Defense  Analyses  (IDA)  began  this  study  in  1991  after  participat¬ 
ing  as  technical  advisors  for  the  SCEs  performed  at  the  National  Test  Facility  (NTF)  and 
on  the  two  Brilliant  Pebbles  (BP)  contractors.  We  applied  the  lessons  learned  from  the  NTF 
and  BP  SCEs  in  subsequent  SCEs  performed  on  offerors  for  Brilliant  Eyes  (BE).  The  addi¬ 
tional  lessons  collected  and  observations  made  during  these  SCEs  are  used  as  the  basis  of 
the  information  presented  in  this  document. 


1.3  Approach 

The  following  steps  were  taken  in  preparation  for  the  analyses  in  this  paper. 

a.  Trained  IDA  personnel  at  SEI. 

SEI  has  developed  and  administers  a  four-day  training  course  for  conducting  an 
SCE.  The  course  introduces  the  software  evaluation  methodology  that  focuses 
on  KPAs  tied  to  the  maturity  model.  SETs  case  studies  and  mock  evaluations 
are  used  to  provide  some  initial  hands-on  experience  to  the  trainees.  Currently, 
five  IDA  research  staff  members  have  completed  training. 

b.  Developed  supplemental  training  materials. 

Based  on  previous  experiences  in  conducting  three  SCEs  for  BMD,  IDA  reex¬ 
amined  SETs  training  course  and  training  materials  for  completeness  and  appli¬ 
cability  to  the  BMD  program.  IDA  developed  additional  training  materials  to 
emphasize  the  information-gathering  aspects  of  conducting  an  SCE  and  to  share 
the  lessons  learned  from  earlier  SCEs.  A  course  was  administered  at  IDA  for  the 
BE  evaluation  team  which  had  been  previously  trained  at  SEI. 

c.  Participated  in  conducting  evaluations. 

IDA  participated  in  the  SCEs  performed  on  the  competing  contractors  for  BE. 
The  IDA  members  of  an  evaluation  team  act  as  technical  advisors,  providing  ad¬ 
ditional  depth  to  the  team. 

d.  Developed  recommended  procedures  and  techniques. 

The  supplemental  training  materials  developed  by  IDA,  and  the  lessons  learned 
by  the  IDA  participants  on  the  NTF,  BP,  and  BE  evaluation  teams,  have  been 
collected  and  are  the  basis  for  this  document. 

1.4  Organization 

Section  1  presents  a  brief  background  of  the  origins  of  the  evaluation  method  and 
the  approach  taken  in  writing  this  paper.  Section  2  describes  the  views  of  the  Ballistic  Mis¬ 
sile  Defense  Organization  (BMDO)  on  SCEs  and  how  these  views  apply  to  the  BMD  pro¬ 
gram.  In  Sections  3,  4,  and  5,  the  activities  that  surround  SCE  are  presented.  The 
appendices  provide  plans,  worksheets,  and  sample  findings  the  evaluation  teams  are  to  use 
in  obtaining  the  SCE  information.  A  list  of  acronyms  and  a  list  of  references  are  provided 
at  the  end. 


2.  BMD  OBJECTIVES  IN  USING  SCEs 


BMDO  wants  to  ensure  that  BMD  software  developers  have  good  software  practic¬ 
es  to  reduce  the  risk  of  software  cost  and  schedule  overruns.  For  this  reason,  SEI’s  process 
maturity  model  and  practices  are  being  used  in  BMD  to  identify  software  process  risks  and 
to  guide  software  process  improvement.  Since  SEI’s  model  is  limited  in  scope,  the  model 
and  practices  will  be  extended  to  help  evaluate  and  monitor  other  areas  of  importance  to 
the  BMD  software  program,  such  as  trusted  software. 

2.1  Use  the  Current  Process  Maturity  Model 

The  BMD  program  will  use  the  ciurent  SEI  process  maturity  model  and  methods 
for  implementing  the  model.  SETs  methods  have  undergone  two  major  changes  in  the  last 
five  years  and  are  still  evolving.  As  SEI  methods  are  updated,  the  BMD  teams  will  be 
retrained  to  use  the  most  recent  version  being  taught  by  SEI. 

The  first  SEI  process  maturity  model  was  documented  in  1987  by  Humphrey  and 
Sweet  [1987].  This  method  was  based  on  a  questionnaire  consisting  of  101  questions.  Since 
1991,  the  method  evolved  into  what  is  known  as  the  “model-based  approach”  which 
includes  goals  and  practices  associated  with  KPAs.  SEI  is  currently  training  people  in  this 
method  and  the  BMDO  SCE  teams  are  using  it  [SEI  1991a]. 

In  August  of  1991,  SEI  released  a  preliminary  draft  of  the  new  model  commonly 
referred  to  as  the  Capability  Maturity  Model  (CMM).  The  draft  CMM  has  undergone  a  pub¬ 
lic  review  and  is  currently  being  revised.  SEI  plans  to  release  the  new  CMM  in  December 
1992  and  update  the  SCE  training  course  by  March  of  1993.  Once  the  method  is  complete 
and  training  provided,  SDI  will  use  the  CMM  for  performing  SCEs. 

2.2  Encourage  Continuous  Process  Improvement 

The  underlying  goal  when  conducting  SCEs  is  to  encourage  continuous  process 
improvement  among  software  development  contractors.  It  should  be  an  on-going  effort 
within  a  contractor’s  organization  rather  than  something  done  only  once.  Continuous  pro¬ 
cess  improvement  will  not  simply  happen  because  a  contractor  or  program  office  wishes  or 


requires  it.  Rather,  a  comprehensive  process  improvement  plan  must  be  put  in  place  and 
followed. 

Due  to  the  large  number  of  contractors  involved  in  the  BMD  program,  it  is  desirable 
to  use  both  the  SCE  and  the  software  process  assessments  (SPA)  to  encourage  continuous 
process  improvement.  SCEs  are  evaluations  performed  by  a  government  team  on  the  con¬ 
tractor’s  process,  whereas  a  SPA  is  performed  by  a  contractor  team  on  its  own  software  pro¬ 
cess.  BMDO  Software  Directive  3405  specifies  an  evaluation  and  assessment  hierarchy,  as 
pictured  below. 


BE  Brilliant  Eyes 

BP  Brilliant  Pebbles 

C2E  Command  Center  Element 

FERDC  Federally  Funded  Research  and  Development  Center 


GBI  Ground  Based  Interceptor 

GBR  Ground  Based  Radar 

PO  Program  Office 

BMDO  Ballistic  Missile  Defense  Organization 


Figure  1.  BMD  Contract  Monitoring  Process 


The  Directive  specifies  that  the  subcontractors  and  the  prime  contractors  perform 
annual  self-assessments  of  their  development  processes  and  develop  annual  software  pro¬ 
cess  improvement  plans.  The  contractors  are  responsible  for  identifying  and  improving 
their  process  over  the  life  of  the  BMD  program.  In  addition,  prime  contractors  are  respon¬ 
sible  for  the  quality  and  cost  of  their  subcontractor’s  software.  Thus,  the  prime  contractors 


are  required  to  perform  annual  SCEs  on  their  subcontractors.  The  BMD  evaluation  team 
will  only  evaluate  prime  contractors  and  not  all  of  the  subcontractors.  But  the  BMD  teams 
will  look  closely  at  how  well  the  prime  contractors  oversee  their  subcontractors  and  will 
validate  the  results  of  the  self-assessments  and  the  quality  of  the  contractor’s  software  pro¬ 
cess  improvement  plans.  The  results  of  the  SCEs  are  provided  to  the  contractor  and  the  ele¬ 
ment  program  managers  for  input  to  their  risk  management  process. 


2.3  Scheduling  SCEs  Throughout  BMD  Life  Cycle 

BMD  program  offices  will  use  SCEs  for  input  to  the  Source  Selection  Evaluation 
Board  (SSEB)  and  to  help  monitor  contracts  previously  awarded.  SCEs  will  be  used  for 

•  source  selection  for  both  DemonstrationA^alidation  (Dem/Val)  and  Engineering  Manufac¬ 
turing  and  Development  (EMD).  Since  it  may  take  one  to  three  years  for  a  contractor  to 
advance  from  one  maturity  level  to  another,  BMD  plans  include  the  use  of  SCEs  as  a  con¬ 
tract  monitoring  mechanism  to  encourage  the  contractors  to  continuously  improve  their 

•  software  development  process,  as  noted  in  Figure  2. 


Start  Start 

DenVV  al  EMD 


1  1-2  yrs 

1-2  yrs 

1-2  yrs 

1-2  yrs 
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1  ^  ^  i 

^  1 

Source 

Contract 

Contract 

Source 

Contract 

Contract 

Selection 

Monitoring 

Monitoring 

Selection 

Monitoring 

Monitoring 

SCE 

SCE 

SCE 

SCE 

SCE 

SCE 

Figure  2.  Schedule  for  SCEs 


The  first  SCE  should  be  performed  during  source  selection,  the  next  SCE  one  to  two 
years  later  to  give  the  contractor  an  opportunity  to  improve  and  to  monitor  his  progress. 

•  Contractors  typically  welcome  SCEs  as  a  contract  monitoring  mechanism  since  it  gives 
them  an  independent  view  of  their  process  and  an  opportunity  to  prepare  for  the  SCE  that 
will  be  used  at  the  next  source  selection.  If,  however,  an  SCE  was  not  performed  during 
source  selection,  an  SCE  should  be  done  approximately  six  to  nine  months  after  the  con- 

•  tract  is  awarded,  as  noted  in  Figure  3.  By  this  time,  the  contractor  should  have  the  software 
process  defined  and  documented  in  a  Software  Development  Plan  (SDP).  It  is  important  to 
have  the  SDP  available  prior  to  an  SCE  so  that  the  evaluation  team  can  verify  that  the  pro- 
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cess  being  described  in  the  interviews  is  the  same  process  being  applied  to  the  BMD  ele¬ 
ment. 


Start  Start 

DenVV  al  HMD 


|6-9  mo 

1-2  yrs 

1-2  yrs 

1-2  yrs 

1-2  yrs 

1  1  i 

Contract  Contract  Source  Contract  Contract 

Monitoring  Monitoring  Selection  Monitoring  Monitoring 
SCE  SCE  SCE  SCE  SCE 

Figure  3.  Alternative  Schedule  for  SCEs 

2.4  Emphasis  of  BMD  Projects  in  SCEs 

When  performing  an  SCE,  the  evaluation  team  looks  at  several  projects  within  the 
contractor’s  organization  to  gain  an  understanding  of  the  contractor’s  software  develop¬ 
ment  process.  The  SCE  team  reviews  information  on  six  to  nine  of  the  contractor’s  projects 
and  interviews  people  from  three  projects.  Section  4. 1  describes  how  the  team  selects  the 
projects  to  review.  The  team  looks  at  several  projects  in  order  to  determine  what  processes 
are  unique  at  the  project  level  and  what  processes  are  standard  across  the  organization  and 
applied  to  all  of  the  projects.  When  new  software  projects  are  initiated,  it  is  desirable  to 
have  a  well-defined  organizational  approach  to  software  process  development  from  which 
a  new  software  project  can  draw.  The  organization’s  standard  software  process  should  be 
derived  and  refined,  based  on  the  experience  and  “lessons  learned”  from  the  projects  within 
the  organization.  It  is  not  desirable  for  each  project  to  learn  the  lessons  first  hand,  but  rather 
to  learn  from  the  trial  and  error  of  previous  projects.  Thus,  an  SCE  evaluates  the  organiza¬ 
tion’s  approach  to  software  development  by  looking  at  the  organizational  practices  and  how 
they  are  used  in  on-going  projects. 

When  a  project  is  initiated,  the  organization’s  standard  approach  to  software  devel¬ 
opment  has  much  more  effect  than  when  a  project  is  nearing  completion.  Consequently,  the 
focus  of  a  BMD  SCE  will  vary,  depending  on  whether  the  BMD  project  is  just  being  initi¬ 
ated  or  whether  it  is  well  into  development.  For  example,  at  the  start  of  DenVVal,  the  off¬ 
erors  will  not  have  a  well-defined  software  process  for  the  BMD  element  for  which  they 
are  competing.  An  SCE  for  DenVVal  source  selection  should  therefore  evaluate  the  pro¬ 
cesses  being  applied  to  other  projects  within  the  contractor’s  organization  supporting 
BMD.  Once  the  DerrVVal  contract  is  awarded,  the  contractor  will  begin  to  define  the  soft- 


ware  process  for  the  BMD  element.  By  the  time  Dem/Val  is  completed,  the  process  being 
applied  to  the  HMD  contract  is  well  defined  for  the  BMD  element.  The  SCE  performed  for 
EMD  source  selection  will  focus  less  on  the  organization’s  process  and  more  on  the  process 
being  used  on  the  BMD  element. 

2.5  Extending  the  Process  Maturity  Model 

The  KPAs  in  SEI’s  Process  Maturity  Model  (PMM)  cover  certain  components  that 
are  recognized  as  part  of  good  software  development  practice.  BMDO’s  efforts  to  improve 
the  development  process,  such  as  the  BMD  Trusted  Software  Development  Methodology 
(TSDM),  raise  additional  requirements  that  could  be  included  in  SCEs.  Other  assessment 
methods  such  as  Software  Development  Capability/Capacity  Review  (SDC/CR)  [AFSC 
1991]  and  Software  Productivity  Research  (SPR)  [SPR  1991]  also  show  that  the  PMM  is 
not  exhaustive  and  could  be  extended  for  BMD  needs.  For  example,  additional  areas  of 
investigation  in  SDC/CR  but  not  found  in  the  SEI  KPAs  are  systems  engineering  and  devel¬ 
opment  tools.  SPR  has  additional  areas  of  coverage  such  as  the  physical  environment,  expe¬ 
rience  of  the  staff,  and  development  methodologies. 

The  basic  approach  for  extending  the  SCE  and  PMM  into  additional  areas  will  be 
to  define  additional  KPAs  for  the  new  requirements.  For  reference,  these  may  be  called  pro¬ 
gram-defined  KPAs,  in  contrast  to  SEI-defined  KPAs  included  in  the  PMM  or  CMM.  In 
adding  program-defined  KPAs,  it  is  most  important  to  preserve  intact  SEI’s  model  and 
assessment  approach  for  several  reasons.  First,  this  allows  a  program  to  leverage  off  SEI- 
developed  training  and  the  experience  of  DoD  and  industry  in  practicing  SETs  approach. 
Second,  a  program  can  use  the  benefit  data  emerging  from  this  experience  as  an  aid  to 
assessing  its  own  improvement  achievements  or  problems.  Third,  a  program  should  find  it 
expeditious  and  economical  to  address  the  added  requirements  with  the  same  team  and  at 
the  same  site  visit  as  the  SEI  evaluation. 

The  addition  of  a  program-defined  KPA,  such  as  for  the  BMD  TSDM,  involves 
developing  a  set  of  explicit  criteria  for  satisfaction  of  the  KPA.  The  criteria  may  be 
expressed  as  direct  requirements  or  as  a  set  of  candidate  questions  to  be  answered  through 
the  SCE  team’s  interview  approach.  The  criteria  also  must  lead  to  a  report  for  the  added 
KPA,  similar  to  the  one  prepared  for  SEI’s  KPAs;  that  is,  it  must  help  identify  strengths  and 
weaknesses,  and  support  a  clear  resolution  of  whether  or  not  the  KPA  is  satisfied. 

Any  score  or  level  scale  associated  with  added  KPAs  should  be  separate  and  distinct 
from  SEI  maturity  levels  (to  preserve  SETs  approach  without  change).  Strengths,  weak- 


nesses,  and  KPA  satisfaction  should  be  the  important  consideration  in  either  case,  not  over¬ 
all  score  or  maturity  level. 

A  program-defined  KPA  may  involve  requirements  in  which  SCE  team  members 
are  not  well  versed.  This  means  that  special  training  may  have  to  be  established  for  SCE 
teams,  so  that  they  can  consistently  judge  various  implementations  of  requirements  as  they 
will  encounter  in  practice. 

Program-defined  KPAs  may  depend  in  part  upon  evidence  and  requirements  estab¬ 
lished  for  SEI’s  KPAs.  An  SCE  team  should  have  this  overlap  clearly  in  mind  during  inter¬ 
views  in  order  to  gather  all  the  pertinent  facts  at  the  most  convenient  time,  rather  than  doing 
repeat  interviews  to  handle  added  KPAs  separately  from  SEI’s  KPAs. 

An  SCE  team  must  be  able  to  address  all  added  KPAs  within  a  reasonable  additional 
time  on  site,  perhaps  no  more  than  one  additional  day.  This  factor  especially  limits  the  num¬ 
ber  and  scope  of  additional  requirements.  Experience  with  added  KPAs  is  needed  to  pro¬ 
vide  firm  guidelines  on  scheduling. 


3.  ACTIVITIES  PRIOR  TO  CONDUCTING  EVALUATIONS 


Prior  to  performing  SCEs,  several  activities  need  to  be  accomplished  so  that  the 
contractor  and  government  program  office  can  incorporate  the  SCEs  into  their  schedules 
and  budgets.  In  order  of  occurrence,  they  are  1)  develop  inputs  to  the  Request  for  Proposal 
(RFP),  2)  establish  evaluation  criteria,  3)  identify  specific  program  needs,  and  4)  select  the 
evaluation  team.  This  section  will  discuss  these  activities  in  more  detail. 

3.1  Develop  Inputs  to  RFP 

When  using  SCEs  for  source  selection  and  contract  monitoring,  the  offerors  must 
be  made  aware  of  the  requirement  in  the  RFP.  Appendix  A  of  this  document  provides  text 
that  can  be  used  to  insert  the  SCE  requirements  notice  within  the  RFP  and  the  associated 
Instructions  for  Preparation  of  Proposals  (IFPP).  The  text  is  expected  to  be  tailored  to 
accommodate  the  specific  requirements  of  the  acquisition. 

Appendices  B  and  C  provide  additional  information  that  should  be  included  in  the 
RFP.  Appendix  B  is  a  sample  project  profile  that  each  contractor  should  complete.  The 
project  profile  requests  general  information  about  a  software  development  effort,  such  as 
coding  language  used,  host  development  system,  and  applicable  standards.  Appendix  C  is 
a  modified  version  of  SEI’s  PMM  questionnaire.  The  questionnaire  requests  detailed  infor¬ 
mation  on  the  software  engineering  practices  used  on  a  project.  Each  of  the  contractors 
should  complete  six  to  nine  of  these  forms.  The  SCE  team  will  then  review  the  forms  to 
select  the  set  of  projects  to  examine  during  the  site  visit. 

3.2  Establish  Evaluation  Criteria 

The  Source  Selection  Authority  (SSA)  will  use  the  results  of  an  SCE  during  source 
selection  as  either  a  specific  criterion  or  a  general  consideration.  The  former  is  preferred, 
since  the  SCE  process  can  provide  valuable  information  useful  in  selecting  a  contractor.  In 
either  case,  the  SSA  will  request  that  evaluation  criteria  be  established  for  using  SCE 
results  to  support  either  a  color  rating  or  a  numerical  rating,  depending  on  whether  procure¬ 
ment  is  for  the  Air  Force  or  the  Army.  Both  rating  systems  should  identify  software  devel- 
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opers  as  high  risk  if  they  have  a  low  process  maturity,  and  low  risk  if  the  maturity  rating  is 
high.  In  other  words,  contractors  with  a  level  1  maturity  should  be  ranked  lower  than  those 
with  a  level  3  maturity.  The  criteria  for  establishing  a  rating  of  the  contractor’s  maturity  is 
based  on  the  findings  in  each  of  the  KPAs. 

For  example,  the  following  criteria  might  be  used  to  map  SCE  results  to  the  Air 
Force’s  blue,  green,  yellow,  and  red  rating  scheme: 

•  Blue:  A  blue  rating  is  given  when  the  SCE  findings  show  that  the  offeror  is 
acceptable  in  all  of  the  following  KPAs:  project  management,  project  planning, 
quality  assurance,  configuration  management,  training,  peer  reviews,  software 
engineering  process  group,  standards  and  procedures. 

•  Green:  A  green  rating  is  given  when  the  SCE  findings  show  the  offeror  is 
acceptable  in  all  of  the  following  KPAs:  project  management,  project  planning, 
quality  assurance,  configuration  management.  In  addition,  the  offeror  is  accept¬ 
able  in  at  least  two  of  the  following  KPAs:  training,  peer  reviews,  software  engi¬ 
neering  process  group,  standards  and  procedures. 

•  Yellow:  A  yellow  rating  is  given  when  the  SCE  findings  show  the  offeror  is 
acceptable  in  at  least  three  of  the  following  KPAs:  project  management,  project 
planning,  quality  assurance,  configxu-ation  management. 

Red:  The  red  rating  is  given  when  the  SCE  findings  show  the  offeror  is  accept¬ 
able  in  fewer  than  three  of  the  following  KPAs:  project  management,  project 
planning,  quality  assurance,  configuration  management. 

As  an  alternative,  the  following  numerical  rating  method  might  be  used.  In  this 
example,  a  total  of  eight  points  can  be  earned  as  follows: 

•  For  each  of  the  following  KPAs  that  are  acceptable,  the  offeror  earns  a  point: 
project  management,  project  planning,  quality  assurance,  configuration  man¬ 
agement. 

•  For  the  offeror  to  earn  any  additional  points,  the  offeror  must  have  been  accept¬ 
able  in  at  least  three  of  the  KPAs  identified  above.  The  offeror  can  earn  an  addi¬ 
tional  point  for  the  following  KPAs,  provided  that  they  were  acceptable: 
training,  peer  reviews,  software  engineering  process  group,  standards  and  pro¬ 
cedures. 
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The  SSEB  will  apply  either  the  color  rating  scheme  or  the  numerical  scheme  to  the 
SCE  results.  The  examples  given  previously  may  be  tailored  to  meet  the  specific  acquisi¬ 
tion  needs.  If  the  program  office  is  selecting  between  two  contractors  that  have  similar 
maturity  levels,  the  criteria  should  be  more  stringent  to  differentiate  between  them. 
Depending  on  the  program  office’s  concerns,  the  evaluation  criteria  can  be  tailored  to 
emphasize  specific  KPAs. 

3.3  Identify  Specific  Program  Needs 

Each  BMD  element  program  office  may  have  specialized  software  needs.  It  is  con¬ 
ceivable  that  the  specialized  needs  of  one  or  more  of  the  elements  would  not  be  appropri¬ 
ately  addressed  in  an  SCE  performed  with  the  standard  PMM-based  KPA  set.  The  program 
office  will  need  to  determine  if  additional  KPAs  need  to  be  added. 

In  addition  to  supplementing  the  list  of  KPAs,  the  program  office  may  also  wish  to 
examine  SEI’s  KPAs  for  ones  of  special  interest  to  the  program  office.  The  SCE  team  would 
then  put  extra  emphasis  on  collecting  data  from  the  contractor  for  that  KPA. 

3.4  Select  the  Evaluation  Team 

When  forming  an  SCE  team,  the  program  office  must  coordinate  through  BMDO 
which  is  responsible  for  SCE  team  training  and  schedules.There  are  several  qualification 
requirements  that  BMDO  requires  of  the  team  and  its  members. 

All  team  members  must  have  attended  an  SCE  training  course,  preferably  together. 
Currently,  SEI  conducts  a  four-day  course.  In  the  future,  other  training  sources,  such  as  the 
Defense  Systems  Management  College,  may  be  available. 

Before  being  selected  for  SCE  training,  potential  trainees  must  have  adequate  soft¬ 
ware  technical  or  managerial  experience.  SEI  recommends  that  trainees  have  at  least  seven 
years  of  software  development  or  acquisition  experience.  It  is  not  adequate  to  have  a  SCE 
team  consist  of  solely  software  acquisition  experts  nor  development  experts.  It  is  desirable 
to  have  a  mix  of  both  professions  on  the  team;  at  least  one  representative  must  have  exten¬ 
sive  software  acquisition  experience  and  at  least  three  representatives  must  have  software 
development  experience.  When  deciding  the  team  composition,  it  is  also  important  that  at 
least  two  of  the  members  have  a  strong  background  in  each  of  SEI’s  key  process  areas. 

BMD  teams  should  not  consist  of  members  from  a  single  Service  or  government 
organization.  Representatives  from  the  Air  Force,  Army,  NTF,  and  Federally  Funded 
Research  and  Development  Centers  (FFRDCs)  should  be  considered  for  the  team. 


Other  considerations  for  selecting  SCE  team  members  include  team  skills,  leader¬ 
ship,  and  lack  of  conflict  of  interest.  Team  skills  are  an  important  aspect  during  an  evalua¬ 
tion.  Those  who  find  the  consensus  process  difficult  and  those  who  are  unable  to  contribute 
to  the  SCE  process  will  not  be  effective  SCE  team  members.  Team  members  must  have 
good  communication  skills  in  order  to  work  with  other  team  members  and  with  contractors 
during  the  evaluation  process.  Team  members  must  be  good  listeners  so  that  they  can  judge 
what  they  hear  during  the  evaluation  process.  Team  members  should  also  take  initiative. 
Without  such  initiative,  the  evaluation  process  might  become  shallow  and  superfluous. 

It  is  also  important  to  have  some  team  members  who  can  take  a  leadership  role  dur¬ 
ing  an  SCE  to  ensure  that  the  SCE  progresses  smoothly  and  effectively.  All  SCE  team 
members  will  be  required  to  sign  a  Procurement  Integrity  Certificate  (PIC)  for  the  source 
selection  at  hand.  The  program  office  should  check  to  ensure  that  all  potential  SCE  team 
members  will  be  able  to  sign  the  PIC.  (Among  other  things,  the  PIC  requires  disclosure  of 
financial  interest  in  any  of  the  RFP  offerors.) 

If  the  chosen  SCE  team  has  little  previous  SCE  experience  or  has  not  worked 
together  on  an  actual  evaluation,  it  is  beneficial  to  arrange  additional  training.  The  purpose 
of  such  training  is  to  sharpen  skills  learned  during  formal  SCE  training  and  to  develop  inter¬ 
personal  team  building  skills.  The  PMM  and  the  SCE  process  discussed  during  the  SCE 
training  course  should  be  reviewed.  In  particular,  each  KPA  should  be  reviewed  to  famil¬ 
iarize  team  members  with  the  criteria  to  be  used  during  the  evaluation.  If  possible,  a  mock 
SCE  should  be  arranged,  consisting  of  interviews  with  “typical”  people  encountered  during 
site  visits.  A  mock  evaluation  helps  build  interpersonal  team  skills  prior  to  the  first  actual 
site  visit. 


4.  CONDUCTING  EVALUATIONS 


This  section  provides  guidance  to  an  SCE  team  for  the  period  between  when  RFP 
responses  are  received  and  the  team  has  completed  the  three  day  on-site  evaluation  of  each 
contractor.  In  order  of  sequence,  the  SCE  team  will  1)  select  projects  to  be  evaluated,  2) 
establish  site  visit  schedule,  3)  submit  documentation  request  to  the  contractor,  4)  develop 
site  visit  entrance  briefings,  and  5)  establish  interview  approach.  This  section  will  provide 
additional  details  on  each  of  these  activities. 

4.1  Select  Projects  To  Be  Evaluated 

In  response  to  the  project  profile  request  in  the  RFP,  a  contractor  will  provide  infor¬ 
mation  on  six  to  nine  projects.  The  SCE  team  will  use  information  contained  in  both  project 
profiles  and  the  SEI  questionnaire  to  select  three  projects  to  be  evaluated  in  greater  detail 
during  the  SCE.  This  selection  process  is  critical  to  the  success  of  an  SCE.  The  projects 
selected  must  provide  data  that  can  be  used  to  judge  the  risk  associated  with  awarding  the 
proposed  project  to  the  contractor.  The  following  paragraphs  provide  guidance  for  selecting 
the  projects. 

Projects  examined  during  an  SCE  must  be  from  the  same  site,  division,  group,  or 
profit  center  as  that  of  the  proposed  project.  These  organizational  divisions  vary  consider¬ 
ably  in  industry.  During  an  evaluation,  the  team  will  be  looking  for  an  organization-level 
set  of  policies  and  procedures  that  are  applied  consistently  across  all  projects.  Specifically, 
the  team  will  seek  to  examine  projects  with  a  common  quality  assurance  (QA),  configura¬ 
tion  management  (CM),  and  software  engineering  process  group  (SEPG).  One  way  to 
determine  whether  the  selected  projects  are  appropriate  for  examination  is  to  trace  the  man¬ 
agement  control  from  each  of  these  functions  up  through  the  organization.  These  lines 
should  converge  at  a  point  also  in  the  management  structure  of  the  proposed  project;  i.e., 
the  QA  manager,  CM  manager,  and  SEPG  managers  should  be  the  same  for  all  the  projects 
being  reviewed. 

Contractor  responsibilities  on  the  selected  projects  should  be  similar  as  those  on  the 
proposed  project.  If  the  contractor  is  the  prime  on  the  proposed  project,  it  is  important  that 
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the  projects  selected  for  examination  represent  examples  where  the  contractor  also  served 
as  the  prime.  If  the  contractor  has  never  been  the  prime  before,  this  is  a  risk  that  must  be 
brought  to  the  attention  of  the  Source  Selection  Authority  (SSA).  The  same  is  true  for  any 
of  the  following  conditions: 

•  Projects  which  were  subcontracted  to  another  contractor  will  not  provide  the 
SCE  team  with  the  information  they  need.  In  subcontract  work,  the  develop¬ 
ment  processes  used  by  the  subcontractor  may  have  been  influenced  and  guided 
by  the  prime  contractor  and  do  not  represent  those  of  the  subcontractor. 

•  Similarly,  the  SCE  team  should  avoid  evaluating  projects  where  most  of  the 
software  was  government  furnished  since  the  team  is  evaluating  the  contractor’s 
development  process  rather  than  the  contractor’s  ability  to  integrate  govern¬ 
ment-furnished  software. 

If  the  proposed  project  involves  a  subcontractor,  the  SCE  team  should  select  at  least 
one  project  for  examination  that  has  included  some  subcontract  work.  This  will  allow  the 
SCE  team  to  evaluate  the  contractor’s  ability  to  assess  and  guide  the  processes  used  by  the 
subcontractor. 

Projects  selected  for  examination  during  an  SCE  should  be  technically  related  to  the 
proposed  project.  For  example,  a  management  information  system  (MIS)  project  may  not 
provide  adequate  information  forjudging  the  risk  involved  with  building  a  sophisticated 
launch  control  system. 

The  scale  of  the  development  effort  on  the  selected  projects  should  be  roughly 
equivalent  to  that  of  the  proposed  project.  Staff  resources  and  lines  of  source  code  are  two 
possible  indicators  of  the  scale  of  the  projects.  Despite  guidance  provided  to  the  contractor, 
the  team  may  still  find  wide  ranging  projects  offered  for  their  consideration. 

It  is  preferable  that  the  projects  selected  for  examination  be  on-going  development 
projects.  At  worst  the  projects  may  be  up  to  six  months  into  the  maintenance  phase.  After 
a  project  is  completed  or  is  in  the  advanced  maintenance  phases,  people,  documentation, 
and  tools  used  in  the  project  tend  to  be  harder  to  locate.  Often  people  cannot  remember 
what  was  done  on  older  projects  or  how. 


4.2  Establish  Site  Visit  Schedule 

SEI  provides  a  strawman  site  visit  schedule  in  the  SCE  training  manual.  This  straw- 
man  has  been  elaborated  to  illustrate  the  breadth  and  depth  of  interview  coverage  desirable 
during  the  three-day  site  visit. 

Table  2  represents  a  suggested  scheme  for  allocating  time  for  interviews  during  a 
site  visit.  The  job  titles  used  in  the  table  are  generic  and  refer  to  typical  areas  of  responsi¬ 
bility.  The  amount  of  time  allocated  to  interview  individuals  in  an  area  is  proportional  to 
the  number  of  KPAs  typically  under  the  responsibility  of  that  individual.  The  SCE  team 
should  use  organizational  charts  and  other  documentation  to  identify  the  actual  titles  and 
names  of  the  individuals  with  the  responsibilities  listed  in  Table  2. 


Table  2.  Allotted  Time  for  Interviews 


Position 

Length  of 
Interview  (hrs) 

Number 

Interviewed 

Project  Managers 

0.50 

2 

Software  Managers 

1.25 

2 

Manager  of  QA 

0.50 

1 

Project  QA 

0.75 

1 

Manager  of  CM 

0.50 

1 

Project  CM 

0.75 

1 

SEPG 

1.25 

1 

Standards 

0.75 

1 

Training 

0.50 

1 

Software  Cost  Manager 

0.50 

1 

Subcontract  Manager 

0.75 

1 

Subcontractor 

1.25 

1 

Developer 

0.75 

1 

Some  functions  may  be  grouped  under  a  single  person,  others  may  be  spread  across 
several  individuals.  Table  3  identifies  the  typical  responsibilities  of  these  key  positions  as 
they  relate  to  the  KPAs.  The  KPAs  are  numbered  to  help  the  SCE  team  members  to  priori- 


tize  the  interview  questions.  If  time  runs  out  during  the  interview,  the  KPAs  with  the  highest 
priority  will  at  least  be  addressed 

Using  the  allocation  from  Table  2,  the  site  visit  schedule  may  look  something  like 
Figures  4,  5,  and  6.  These  schedules  allow  fifteen-minute  breaks  between  interviews.  This 
time  can  be  used  to  discuss  findings,  modify  an  interview  approach,  check  documentation 
or  organization  charts,  modify  the  schedule  for  the  day,  or  take  a  necessary  break.  The  SCE 
team  should  avoid  overscheduling.  Always  prioritize  the  list  of  people  to  interview  and  the 
topics  to  be  covered  during  each  interview.  Allow  extra  time  in  the  schedule  to  accommo¬ 
date  unanticipated  interviews  or  document  reviews. 


Table  3.  Topic  Priorities  for  Interviews 


Position 

Priorities  of  KPAs^ 

PM 

PP 

CM 

QA 

ST 

PR 

TR 

SEPG 

Project  Managers  (2) 

1 

2 

3 

5 

4 

6 

SAV  Managers  (2) 

3 

1 

7 

5 

4 

2 

6 

Manager  of  QA 

1 

2 

3 

4 

Project  QA 

1 

2 

3 

4 

Manager  of  CM 

1 

2 

3 

Project  CM 

1 

2 

3 

SEPG 

4 

2 

3 

1 

Standards 

^3 

1 

2 

Training 

1 

2 

S/W  Cost  Manager 

1 

3 

2 

Subcontract  Manager 

1 

2 

6 

3 

5 

4 

Subcontractor 

3 

1 

7 

5 

4 

2 

6 

Developer 

6 

5 

3 

4 

1 

2 

a.  PM  -  Program  Management;  PP  -  Project  Planning;  CM  -  Configuration  Management;  QA  - 
Quality  Assurance;  ST  -  Standards  &  Procedures;  PR  -  Peer  Reviews;  TR  -  Training;  SEPG  - 
Software  Engineering  Processing  Group. 


Figure  4  provides  the  sample  schedule  for  the  first  day  of  the  site  visit.  The  site  visit 
begins  with  entrance  briefings  by  both  the  evaluation  team  and  the  contractor.  The  remain¬ 
der  of  the  day  is  spent  conducting  interviews  with  the  senior  project  personnel. 


8:30  -  9:00 

BMDO  Introductory  Brief 

9:00  -  10:00 

Contractor  Entrance  Briefing 

10:00-11:00 

Documentation  Review 

11:00-5:30 

Interviews: 

1.25  hrs  S/W  Lead  (project  A) 

0.25  Break 

0.50  hr  Program  Manager  (project  A) 

0.25  Break 

0.75  hr  SEPG  Manager 

0.25  Break 

1.25  hrs  S/W  Lead  (project  B) 

0.25  Break 

0.75  hr  S/W  QA  (project  A) 

7:30  -  9:30 

Caucus 

Note:  Lunch  around  12:00  (Ihr) 

Figure  4.  Schedule  for  Day  One 


The  second  day  of  the  site  visit  involves  interviewing  personnel  responsible  for  spe¬ 
cific  KPAs  such  as  standards  and  cost  estimation.  Substantial  time  is  also  allotted  for  doc¬ 
umentation  review.  Refer  to  Figure  5  for  additional  details. 

The  third  day  of  the  site  visit  is  reserved  primarily  for  reviewing  specific  documents 
requested  during  the  course  of  the  interviews  and  for  finalizing  and  documenting  the  SCE 
findings.  Refer  to  Figure  6  for  details. 

4.3  Submit  Documentation  Requests 

Organization  and  project  documentation  is  requested  at  three  stages  for  an  SCE:  pri¬ 
or  to  a  site  visit,  at  the  start  of  a  site  visit,  and  during  the  interviews. 

Prior  to  arrival  on  site,  the  SCE  team  will  have  requested  and  received  a  Software 
Development  Plan  (SDP),  project  profiles,  an  organization  chart,  and  a  completed  SEI 
questionnaire.  The  SDP  should  be  either  generic,  a  corporate  standard  SDP,  or  one  from  the 
projects  offered  for  examination.  The  team  may  request  that  additional  documentation  be 


provided  upon  arrival.  This  documentation  is  used  to  plan  an  interview  schedule,  identify 
issues  and  questions,  and  form  a  basis  for  findings.  Figure  7  lists  the  documents  typically 
requested  for  use  during  the  site  visit. 


8:30  -  10:30 

Interviews: 

0.75  hr  Standards  &  Training 

0.50  hr  S/W  Management  &  Costing 

0.75  hr  S/W  Configuration  Management  (project  B) 

10:30  -  3:00 

Documentation  Review  and  lunch 

0.50  hr  Project  Director  (project  C) 

1.25  hr  S/W  Lead  (project  C) 

0.25  Break 

1 .25  hrs  Subcontractor  S/W  Lead 

0.25  Break 

3:00  -  5:30 

Additional  interviews  (TBD) 

7:30  -  9:30 

Documentation  Review  (at  hotel) 

Note:  Lunch 

around  12:00  (1  hr) 

Figure  5.  Schedule  for  Day  Two 


8:00  -  9:00 

Documentation  Review 

9:00  - 12:00 

Interviews  as  needed 

12:00  -  3:30 

Prepare  Exit  briefing*  (or  Final  Report) 

3:30  Exit  Briefing 

(*None  for  source  selection) 

Figure  6.  Schedule  for  Day  Three 


During  a  site  visit  the  team  may  request  additional  documentation.  These  requests 
come  as  a  result  of  information  gained  during  the  interview  process.  Information  contained 
in  these  documents  will  be  used  to  support  information  learned  during  the  interview,  and 
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may  be  used  to  corroborate  findings  in  the  final  report.  Sample  documents  to  collect  during 
the  interview  process  are  listed  in  Figure  8. 

Project  Documents:  Program  Management  Plan 

Software  Development  Plan 
Software  Configuration  Management  Plan 
Software  Quality  Assurance  Plan 
Software  Test  Procedures 
Software  Standards  and  Procedures  Manual 
Sample  Software  Development  Folder 
Division  Documents:  Software  Policy,  Standards,  and  Procedures 

Generic  Software  Development  Plan 
Quality  Assurance  Plan 
Configuration  Management  Plan 

Figure  7.  Documents  To  Request  Upon  Arrival 


4.4  Develop  Entrance  Briefing 

At  the  beginning  of  the  site  visit,  both  the  evaluation  team  and  the  contractor  pro¬ 
vide  an  entrance  briefing  which  helps  to  identify  the  scope  of  the  SCE  and  the  contractor’s 
software  process. 

The  evaluation  team’s  entrance  briefing  is  given  to  the  contractors  on  the  first  morn¬ 
ing  of  the  site  visit  The  purpose  of  the  briefing  is  to  describe  the  SCE  process  and  what  the 
contractor  can  expect  over  the  next  three  days.  An  annotated  standard  BMD  entrance  brief¬ 
ing  is  available  [Springsteen  1991].  The  briefing  should  inform  the  contractor  that  the  SEI 
questionnaire  is  used  by  the  team  only  to  become  acquainted  with  the  contractor  and  its  pro¬ 
cesses,  and  that  the  SCE  findings  will  be  based  on  information  collected  during  the  site  vis¬ 
it. 

The  contractor’s  entrance  briefing  is  an  opportunity  for  the  SCE  team  to  become 
acquainted  with  the  contractor’s  organization  and  begin  to  collect  information.  The  direc¬ 
tion  the  team  provides  to  the  contractor’s  point  of  contact  will  determine  the  quality  and 
amount  of  useful  information  they  receive  during  the  entrance  briefing.  It  is  important  that 
the  contractor’s  entrance  briefing  not  concentrate  on  findings  from  other  SCEs  or  SPAs.  A 


Project  Management  Documents 

•  Progress  tracking  reports  or  software  status  reports 

•  Subcontractor  status  reports 

•  Policy  and  procedures  for  monitoring  subcontractors 

Project  Planning  Documents 

•  Metrics  reports  (size,  quality,  progress,  computer  performance) 

•  Estimation  process  (size,  schedule,  cost) 

•  Policy  for  committing  or  approving  estimates 

•  Description  of  central  estimation  database 

Quality  Assurance  Documents 

•  Audit  checklists  and  schedules 

•  S  ubcon tractor  audit  reports 

•  Summary  reports  to  senior  management  (e.g.,  non-concurrence  reports) 

Configuration  Management  Documents 

•  Software  configuration  management  plan 

•  Requirements  traceability  matrix 

•  Change  control  board  membership  and  minutes 

•  Version  description  document 

Peer  Review  Documents 

•  Checklist  and  schedules  (design,  code,  and  test  case) 

•  Summary  statistics  (e.g.,  type  of  errors  found  per  life  cycle  phase) 

•  Policy  and  procedures 

•  Minutes  (design,  code,  test  case) 

Training  Documents 

•  Training  policy 

•  Training  records 

•  Schedule 

Standards  Documents 

•  Templates  (SDP,  CM,  QA) 

•  Practices  for  submitting  revisions  to  standards 

•  Software  Standards  and  Procedures  Manual 

SEPG  Documents 

•  Process  Improvement  Plan 

•  Policy  and  procedures  (e.g.,  charter) 


Figure  8.  Documents  To  Request  During  Interviews 
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contractor  could  use  this  information  to  try  to  influence  the  SCE  team.  Since  time  is  limited, 
the  SCE  team  should  ensure  the  entrance  briefing  contains  as  much  useful  information  as 
possible. 

A  significant  amount  of  information  can  be  learned  from  a  review  of  the  contrac¬ 
tor’s  organization  charts.  This  information  can  help  the  team  plan  their  strategy  for  the 
interviews  by  identifying  specific  offices  or  individuals  responsible  for  key  management  or 
technical  functions.  The  team  should  specifically  request  that  the  briefing  include  a  discus¬ 
sion  of  the  lines  and  scope  of  authority  represented  by  the  organization  charts.  This  infor¬ 
mation  indicates  the  extent  to  which  policies  and  procedures  are  institutionalized.  It  may 
also  begin  to  indicate  inhibitors  to  process  improvement. 

Organization  charts  also  serve  as  a  mechanism  to  discuss  the  relationships  between 
projects  within  the  company.  Shared  lines  of  authority,  services,  and  resources  among 
projects  may  indicate  corporate- wide  institutionalization  of  some  policies  or  procedures. 

Support  services  and  groups  are  generally  only  visible  on  higher  level  organization 
charts.  The  organization  charts  may  also  identify  other,  maybe  external,  review  or  approval 
authorities  such  as  a  Risk  Board. 

Lastly,  organization  charts  also  indicate  the  relationship  between  the  contractor  and 
any  subcontractors  involved  with  the  project. 

During  a  discussion  of  the  organization  charts,  the  team  should  identify  or  ask 
where  the  SEPG,  CM,  QA,  Costing,  Standards  and  Procedures,  and  Training  managers  are 
in  the  organization  hierarchy.  These  are  critical  functions.  The  team  may  want  to  suggest 
or  request  a  short  briefing  by  each  of  these  managers.  The  team  can  make  a  formal  request 
for  specific  information,  possibly  to  include  organizational  charts,  in  these  briefings.  The 
information  sought  includes  the  roles  and  responsibilities  of  individuals,  the  scope  and 
influence  of  their  function  on  the  projects  being  examined,  the  resources  under  their  con¬ 
trol,  and  the  products,  standards,  and  tools  they  provide  to  the  rest  of  the  organization. 

4.5  Establish  Interview  Approach 

The  site  visit  allows  the  SCE  team  an  opportunity  to  assess  and  verify  the  software 
practices  being  used  by  the  contractor.  During  the  site  visit,  interviews  are  conducted  with 
project  personnel  and  project  documentation  is  reviewed  to  verify  the  adequacy  of  the  prac¬ 
tices  being  employed  by  the  contractor.  This  section  describes  the  approach  taken  by  the 
SCE  team  to  conduct  and  document  the  interviews. 


4.5.1  SCE  Checklist 


To  ensure  consistency  among  BMD  SCEs,  a  standard  checklist  has  been  developed 
(Appendix  D).  During  the  course  of  the  interviews,  the  checklist  offers  the  SCE  team  a 
quick  reference  to  ensure  that  all  appropriate  topics  are  covered.  The  checklist  identifies  the 
KPAs  and  specific  issues  being  explored  by  the  team.  Federal  Acquisition  Regulations 
require  evaluations  to  be  consistent  [FAR  1992].  Hence,  the  checklist  can  not  be  changed 
between  SCEs. 

4.5.2  Team  Member  Roles 

The  SEI  SCE  training  materials  provide  guidance  on  the  roles  of  SCE  team  mem¬ 
bers  during  a  site  visit.  This  section  includes  additional  roles  created  from  the  lessons 
learned  in  seven  recent  SCEs.  If  possible,  roles  should  be  rotated  during  subsequent  evalu¬ 
ations  in  order  to  increase  the  experience  of  each  team  member. 

a.  Door  Keeper. 

This  person  has  the  responsibility  of  ensuring  that  the  interviews  proceed  unin¬ 
terrupted.  The  door  keeper  should  arrive  at  the  site  with  signs  that  indicate, 
“SCE  Interviews.  Please  Do  Not  Disturb.”  These  signs  should  be  posted  for  the 
duration  of  the  visit.  The  door  keeper  will  ensure  that  the  doors  to  the  interview 
room  remain  closed  during  the  interview  and  will  escort  an  interviewee  in  or  out 
of  the  room. 

b.  Introducer. 

This  person  will  introduce  the  team  members  to  the  contractor  and  give  any  in¬ 
troductory  remarks.  The  introducer  should  reiterate  to  the  contractor  that  all  in¬ 
formation  will  be  kept  in  strict  confidence  and  that  all  comments  made  during 
interviews  will  remain  non-attributed. 

c.  Document  Tracker. 

This  person  wiU  keep  a  log  of  aU  documents  requested  during  the  interviews. 
The  log  should  include  the  following  information:  name  of  interviewee,  docu¬ 
ment  name,  who  requested  it,  associated  KPA,  delivered  (check),  reviewed 
(check).  All  documents  provided  to  the  SCE  team  during  the  visit  should  be 
logged,  distributed,  and  maintained  by  the  document  tracker. 


d.  Consensus  Builder. 

This  individual  ensures  that  each  team  member  is  afforded  the  opportunity  to 
express  his  or  her  opinion  on  a  given  issue.  The  consensus  builder  will  focus  the 
team  during  discussions,  possibly  by  suggesting  what  additional  information 
may  help  the  group  reach  consensus. 

e.  Time  Keeper. 

This  individual  keeps  track  of  the  time  during  the  interviews  and  the  consensus 
meetings  and  is  responsible  for  keeping  the  SCE  team  on  schedule. 

f.  Report  Organizer. 

This  person  maintains  the  final  reports  and  all  the  supporting  documentation. 
4.5.3  Document  Interviews 

It  is  important  that  all  information  gathered  during  the  site  visit  be  documented  so 
that  it  may  later  be  used  to  support  findings.  During  the  site  visit,  interview  notes  serve  to 
stimulate  further  questions  and  documentation  requests  and  to  build  consensus.  Since  time 
is  limited,  the  site  visit  must  be  conducted  efficiently.  Experience  has  shown  that  an  inter¬ 
view  note  template  may  help  organize  the  note  taker’s  thoughts  and  better  support  the  con¬ 
sensus  building  process.  Figure  9  contains  a  template  which  is  convenient  for  formatting 
interview  notes.  In  addition,  the  following  suggestions  are  recommended: 

a.  Use  a  new  notebook  for  each  contractor. 

By  doing  this  the  team  avoids  the  possibility  of  confusing  one  contractor’s  find¬ 
ings  with  another  and  illegally  sharing  information  among  the  competitors. 

b.  Use  a  new  page  for  each  person  interviewed. 

Clearly  identify  the  interviewee  at  the  top  of  the  page.  This  will  eliminate  any 
confusion  regarding  who  may  have  provided  what  information. 

c.  Use  the  KPAs  to  organize  the  information  provided  during  the  interview. 

By  using  the  name  of  KPAs  as  labels,  information  will  be  easy  to  find  later  dur¬ 
ing  team  discussions. 

d.  List  all  the  documents  requested  during  the  interview  in  a  box  at  the  bottom  of 
the  page. 
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The  document  tracker  can  then  do  a  quick  check  before  requesting  information 
from  the  contractor. 

e.  List  questions  that  arise  during  the  interview  in  a  box  at  the  top  right-hand  cor¬ 
ner  of  the  page. 

If  one  team  member  is  leading  the  questioning,  it  is  not  appropriate  for  other 
team  members  to  interrupt  to  ask  their  pressing  questions.  Rather  than  miss  an 
opportunity  to  pursue  an  issue,  document  the  questions  as  they  come  to  mind  so 
they  can  be  asked  at  a  more  appropriate  time. 


Questions 

Name,  title  of  interviewee 

Name  of  KPA  -  Relevant  notes  during  interview 

Requested  documents 

Name  of  document,  who  requested  it,  associated  KPA 


Figure  9.  Template  for  Interview  Notes 
4.5.4  Daily  Objectives 

To  ensure  the  team  allocates  their  time  appropriately  during  the  three-day  site  visit, 
the  following  objectives  must  be  achieved  each  day; 

a.  Day  One. 

The  SCE  team  should  revisit  the  schedule  following  the  first  day  of  interviews. 
Names  of  people  to  be  interviewed  may  need  to  be  added  or  removed  from  the 
schedule  for  the  succeeding  days.  Time  allocations  may  need  to  be  adjusted 
based  on  information  learned  during  the  first  day  of  interviews.  A  list  of  all  the 
documentation  requested  but  not  yet  received  by  the  team  should  be  provided 
to  the  contractor  point  of  contact  at  the  end  of  day  one. 

After  revisiting  the  schedule,  the  SCE  team  should  gather  in  the  evening  to  dis¬ 
cuss  the  results  from  the  first  day  of  interviews.  A  rough  pass  through  each  KPA 
should  be  performed,  writing  down  initial  thoughts  and  impressions.  These  im¬ 
pressions  should  indicate  where  the  contractor  appears  to  be  strong  or  weak.  All 
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conclusions  must  be  corroborated  by  information  provided  by  either  another  in¬ 
terviewee  or  by  documentation.  The  SCE  team  should  plan  how  they  will  un¬ 
cover  additional  information  to  substantiate  their  initial  conclusions. 

b.  Day  Two. 

Following  the  contractor  interviews  on  day  two,  the  SCE  team  should  continue 
to  develop  initial  impressions  and  findings  on  the  strengths  and  weaknesses  of 
the  contractor.  The  team  may  need  to  readjust  the  site  schedule  based  on  infor¬ 
mation  gathered  on  day  two.  For  this  reason,  the  SCE  team  should  prioritize  the 
KPAs  that  still  need  to  be  addressed  before  the  end  of  the  site  visit. 

At  the  end  of  day  two,  each  member  of  the  SCE  team  should  write  a  preliminary 
summary  of  the  strengths  and  weaknesses  for  each  KPA.  The  following  morn¬ 
ing,  the  team  may  review  initial  findings  to  ensure  that  sufficient  information  is 
being  gathered. 

c.  Day  Three. 

At  the  beginning  of  day  three,  the  SCE  team  should  review  each  KPA  individ¬ 
ually.  This  should  be  done  in  a  round  robin  fashion.  The  discussion  should  en¬ 
able  each  team  member  to  provide  feedback  on  his/her  understanding  of  the 
information  gathered  during  the  visit  from  interviews  and  documents.  The  team 
should  identify  areas  of  agreement  and  disagreement  within  the  team,  and 
schedule  the  appropriate  interviews  to  resolve  differences  of  opinion.  The  team 
should  then  make  a  formal  request  to  conduct  these  additional  interviews  with 
the  contractor  point  of  contact. 

All  interviews  must  be  concluded  before  noon  on  the  third  day.  This  will  allow 
sufficient  time  for  the  team  to  prepare  its  findings.  Before  the  team  departs  from 
the  contractor’s  facility  at  the  end  of  the  third  day,  the  SCE  team  should  com¬ 
plete  the  report  of  its  findings. 

4.5.5  Other  Interview  Practices 

Following  are  other  good  practices  for  performing  SCEs. 

a.  Record  information  from  document  review. 

During  any  document  review  phase,  detailed  information  about  the  document 
must  be  kept.  The  individual  reviewing  a  document  should  log  the  title  and  date 
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of  the  document  in  their  notebook.  If  a  process  is  identified  in  the  documenta¬ 
tion,  this  should  be  logged  in  too. 

a.  Establish  privacy  for  SCE  team. 

The  SCE  team  requires  privacy  during  the  interview  process.  The  SCE  team  co¬ 
ordinator  should  request  a  soundproof  room  with  locking  doors  for  use  during 
the  site  visit.  A  room  with  non-soundproof  walls  will  not  permit  free  discussion 
among  the  SCE  team  members. 

b.  Use  Watts  Humphrey’s  book. 

The  members  of  an  SCE  team  should  use  the  SCE  tutorial  information  and 
Humphrey’s  book,  Managing  the  Software  Process  [Humphrey  1989],  as  refer¬ 
ence  material  during  a  site  visit.  This  material  contains  valuable  information, 
and  can  often  clarify  any  confusion  among  team  members.  Each  team  member 
should  bring  this  material  to  each  site  visit  made  by  the  SCE  team. 

c.  Be  aware  of  terminology  differences. 

Terms  may  be  understood  by  people  to  have  different  meanings.  An  example  of 
this  is  the  term  “peer  review.’’  Even  commonly  understood  terms  may  be  used 
by  a  contractor  in  an  entirely  different  way.  The  SCE  team  must  be  flexible  in 
conversing  with  a  contractor.  Make  sure  to  understand  the  contractor’s  lan¬ 
guage  rather  than  insisting  that  the  contractor  use  the  terms  familiar  to  the  team. 

d.  Conduct  interviews  with  the  entire  SCE  team  present. 

Although  time  is  limited  during  the  SCE  site  visit,  it  is  critical  that  the  team  re¬ 
main  together  and  conduct  every  interview  with  all  the  team  members  present. 
It  is  tempting  to  break  the  team  into  subgroups  to  conduct  interviews  in  parallel. 
When  building  consensus,  the  team  will  need  input  from  every  member.  Thus, 
it  is  vital  that  all  interviews  occur  as  a  group. 

e.  Assign  KPAs. 

Several  approaches  can  be  taken  in  assigning  KPA  responsibility  during  the  in¬ 
terviews.  The  team  may  decide  that  each  member  will  assume  primary  respon¬ 
sibility  for  one  KPA,  and  secondary  responsibility  for  another  KPA.  In  this  way 
each  KPA  will  have  the  explicit  attention  of  two  team  members,  although  all 
team  members  will  provide  input  on  that  KPA  during  the  consensus  building 
process.  The  team  member  with  primary  responsibility  for  a  KPA  will  formu- 


late  interview  questions  and  follow-ups,  and  research  the  documentation  for  in¬ 
formation.  This  team  member  may  also  be  responsible  for  producing  the 
wording  for  the  findings  slides  and  final  report.  The  team  member  with  second¬ 
ary  responsibility  will  also  pursue  information  on  the  KPA  and  provide  back-up 
during  the  interview  process. 

Another  approach  is  to  allow  team  members  to  pursue  all  KPAs  for  which  they 
feel  qualified. 

f.  Take  advantage  of  breaks  between  interviews. 

The  breaks  between  interviews  can  be  effectively  used  by  the  SCE  team.  Breaks 
provide  an  opportunity  for  the  team  members  to  discuss  information  gathered 
during  the  last  interview  and  information  remaining  to  be  confirmed  or  pursued 
during  the  next  interview.  Breaks  can  also  be  used  by  the  team  members  to  dis¬ 
cuss  the  interview  process,  a  team  member’s  approach  to  seeking  specific  infor¬ 
mation,  or  to  clarify  or  improve  the  effectiveness  of  questions  being  asked. 

g.  Establish  consensus. 

During  the  consensus-building  process,  the  team  should  listen  to  each  member. 
The  Consensus  Builder  is  responsible  for  ensuring  that  this  occurs.  Avoid  trying 
for  force  consensus  too  early  while  information  is  still  being  collected.  Verify 
all  conclusions  by  cross-checking  with  other  team  members’  interview  notes 
and  with  documents  provided  by  the  contractor. 

It  is  important  to  determine  early  if  there  are  different  opinions  on  whether  the 
contractor  is  satisfying  a  KPA.  If  consensus  is  not  established,  this  indicates  that 
more  information  is  needed.  Identify  what  information  is  necessary  to  support 
the  different  view  points  and  appropriately  adjust  the  interview  schedule  and 
document  requests  list.  Consensus  must  be  established  when  ranking  the  KPAs 
as  acceptable  or  unacceptable.  It  is  less  critical,  however,  to  have  consensus  on 
the  contractor’s  individual  strengths  and  weaknesses  for  particular  KPAs.  In 
this  event,  the  team  should  reach  a  compromise  on  the  wording  of  the  strength 
or  weakness  to  accurately  reflect  the  contractor’s  process. 


5.  ACTIVITIES  AFTER  CONDUCTING  AN  EVALUATION 


This  section  of  the  report  will  review  the  activities  that  occur  after  the  SCE  site  visit. 
Included  in  this  section  is  a  description  of  the  final  SCE  report  and  the  feedback  to  the  con¬ 
tractor,  the  SSEB,  the  program  office,  and  BMDO  concerning  the  results  of  the  SCE. 

5.1  Final  Report 

The  SCE  team  must  prepare  a  report  of  its  findings  and  their  rationale  for  the  con¬ 
tractor,  SSEB,  program  office,  and  BMDO.  Refer  to  Figure  10  for  a  depiction  of  the  outline 
of  the  report. 


1.  Summary  of  Findings 

•  Acceptable  KPAs 

•  Unacceptable  KPAs 

•  Summary  of  contractor’s  software  process  improvement  plans 

2.  Summary  of  Project  Information 

•  List  of  original  projects  submitted  (7-9  projects) 

•  List  of  projects  selected  (1-3  projects) 

•  Rationale  for  project  selection 

•  Interview  schedule,  interviewee  names  and  positions 

3.  Details  of  KPAs 

•  Exit  briefing  slides 

•  Rationale  for  each  bullet 

Appendix 

•  Project  Profiles  (7-9  projects) 

•  Supporting  documentation 


Figure  10.  Report  Outline 

The  first  section  of  the  SCE  report  should  include  a  high-level  summary  of  the  find¬ 
ings.  A  list  of  the  acceptable  and  unacceptable  KPAs  should  be  provided  along  with  an 
overview  of  the  contractor’s  software  process  improvement  plans. 


The  second  section  should  describe  the  focus  of  the  site  visit.  It  should  include  a 
summary  of  the  SEI  questionnaires  and  a  summary  of  the  six  to  nine  project  profiles  sub¬ 
mitted  by  the  contractors.  In  addition,  the  site  visit  schedule  should  be  provided  along  with 
the  rationale  for  selecting  the  specific  projects  to  review. 

The  main  body  of  the  report  will  contain  the  exit  briefing,  annotated  to  include  the 
details  of  each  of  the  KPAs.  Thus,  the  main  body  of  the  report  should  include  a  description 
of  each  bullet  that  appears  in  the  exit  briefing  (Figure  8)  so  that  the  SSEB  and  the  contrac¬ 
tors  may  fully  understand  each  finding. 

Each  of  the  slides  should  contain  a  summary  statement  of  the  KPA,  and  a  list  of 
KPA  strengths,  weaknesses,  and  improvement  activities  the  contractor  has  planned.  The 
summary  statement  of  the  KPA  should  identify  whether  the  KPA  was  acceptable  or  unac¬ 
ceptable  and  the  prevailing  rationale.  This  is  the  most  important  section  of  the  report.  Refer 
to  the  example  below  (Figure  11). 

Peer  Reviews 

Summary:  Peer  Review  process  is  unacceptable,  the  process  is  informal 

and  not  applied  consistently  across  all  projects. 

Strengths: 

•  Some  projects  perform  low-level  inspections  of  critical  modules 

Weaknesses: 

•  Design,  code,  unit  test  cases  not  consistently  reviewed  across  projects 

»  No  formal  procedures  for  conducting  peer  reviews  (e.g.,  checklist) 

•  Lack  of  formal  reporting  and  tracking  of  peer  review  findings 

•  Lack  of  statistics  on  findings,  results,  and  effort 

Planned  Improvement  Activities 

•  None 

Figure  11.  Sample  Exit  Briefing  Slide 

Appendix  E  contains  a  list  of  strengths  and  weaknesses  for  each  of  the  KPAs  used 
by  previous  BMD  teams.  The  SCE  team  can  use  these  to  help  formulate  the  wording  of 
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their  findings.  It  is  important,  however,  to  use  only  the  findings  that  have  been  verified 
through  the  interview  process  or  documentation  reviews. 

The  appendix  of  the  report  should  contain  substantiating  information  provided  by 
the  contractors  that  support  the  findings.  Included  in  this  section  are  the  six  to  nine  project 
profiles  and  questionnaires  submitted  by  the  contractors.  In  addition,  copies  of  information 
that  support  a  finding  should  be  collected  in  the  event  that  the  SSEB  requires  additional 
detail.  For  example,  a  page  of  the  SDP  that  identifies  details  of  an  unsatisfied  KPA  may  be 
copied  and  included  in  the  appendix  for  future  reference. 

The  report  must  be  marked  CONFIDENTIAL  and  distribution  controlled.  The 
report  should  be  made  available  only  to  the  SSEB,  the  element  program  manager,  element 
software  lead,  and  the  Director  of  Computer  Resources  Engineering  at  BMDO. 

5.2  Contractor  Feedback  of  Evaluation  Results 

The  contractor  may  provide  feedback  on  the  SCE.  When  SCEs  are  used  for  source 
selection,  the  competing  contractors  will  not  receive  an  exit  briefing  at  the  end  of  the  SCE 
site  visit  Instead,  the  winning  contractors  will  be  briefed  soon  after  the  contract  is  awarded. 
At  that  time  the  winning  contractors  have  an  opportunity  to  provide  feedback  of  the  SCE 
results  and  to  clarify  any  questions.  The  losing  contractors  will  receive  a  very  high-level 
overview  of  the  SCE  results  at  the  “loser’s  conference,’’  and  no  detailed  feedback  will  be 
provided  of  the  SCE  results  at  this  time. 

When  SCEs  are  used  to  help  monitor  a  contract,  the  contractors  will  receive  a 
detailed  Exit  Briefing  at  the  end  of  the  SCE  site  visit.  The  Exit  Briefings  offers  the  contrac¬ 
tor  an  opportunity  to  understand  the  SCE  results  and  to  question  the  team  on  its  findings. 
Since  the  Exit  Briefing  slides  are  not  very  detailed  and  the  audience  is  generally  very  large, 
contractors  may  request  a  follow-up  meeting  with  their  SEPG  and  the  SCE  team  to  ensure 
that  the  findings  are  interpreted  correctly  and  incorporated  into  their  software  process 
improvement  plans. 

5.3  Use  of  Evaluation  Results  in  Source  Selection 

One  of  the  most  important  program  management  responsibilities  in  using  an  SCE 
for  source  selection  is  defining  how  the  SCE  results  should  be  communicated  to  the  SSEB. 
The  SCE  team  and  the  SSEB  must  agree  upon  the  format.  One  method  that  has  been  used 
successfully  is  for  the  SCE  team  to  provide  the  SSEB  a  copy  of  the  SCE  report  after  each 
site  visit.  The  report  contains  a  summary  of  the  contractor’s  strengths,  weaknesses,  and 


improvement  plans  for  each  KPA.  In  addition  the  summary  identifies  each  KPA  as  accept¬ 
able  or  unacceptable.  Refer  to  Section  5.1  for  additional  details  of  the  final  report  format. 

This  summary  of  the  KPAs  is  used  by  the  SSEB  and  is  incorporated  with  the  rest  of 
the  source  selection  process.  Refer  to  Section  3.2  for  details  of  the  criteria  the  SSEB  will 
use  to  assign  a  color  or  numerical  rating.  In  the  event  that  the  SSEB  requires  additional  clar¬ 
ification  of  the  findings,  the  SCE  team  leader  should  be  available  to  respond  to  any  ques¬ 
tions. 

5.4  Use  of  Evaluation  Results  for  Contract  Monitoring 

When  an  SCE  is  performed  to  help  monitor  the  contract,  the  program  office  should 
compare  the  results  of  the  evaluation  with  the  contractor’s  process  improvement  plans.  If 
there  is  a  discrepancy  between  the  weaknesses  identified  by  the  independent  SCE  team  and 
those  of  the  contractor’s  self-assessment,  the  contractor  should  be  made  aware  of  the  dif¬ 
ferences.  The  program  office  should  ensure  that  the  contractor  has  an  acceptable  plan  to 
mitigate  the  risks  that  could  result  from  the  weaknesses  and  to  reduce  the  principal  weak¬ 
nesses  over  the  length  of  the  contract. 

There  are  several  approaches  the  program  office  can  take  to  encourage  contractors 
to  improve  their  software  development  processes.  One  is  to  emphasize  to  the  contractor  the 
importance  of  software  process  improvement  in  the  BMD  program.  If  there  are  two  or  more 
contractors  competing  for  a  future  contract,  the  program  office  can  state  its  plans  to  use 
SCE  results  during  the  next  source  selection  process.  This  should  help  to  motivate  the  com¬ 
peting  contractors  to  improve  so  that  they  can  better  their  scores  during  the  next  source 
selection.  In  addition,  the  contractors  can  report  the  status  of  their  process  improvement 
program  at  the  software  reviews  (e.g.,  preliminary  design  review  or  critical  design  review). 
At  the  program  reviews,  the  contractors  should  be  required  to  describe  their  improvement 
plans  and  accomplishments. 

Another  means  of  encouraging  contractors  to  improve  is  to  incorporate  software 
process  improvement  into  an  award-fee  program.  Since  process  improvement  focuses  on 
the  organization  rather  than  on  a  specific  program,  not  all  projects  are  good  candidates  for 
an  award  fee.  The  best  are  those  contractors  who  have  an  organization  devoted  to  the  BMD 
program  and  are  working  on  several  BMD  projects.  For  example,  the  National  Test  Bed  and 
Integration  Contractor  (NTBIC)  would  be  a  suitable  candidate. 

If  an  award  fee  is  suitable,  the  award  fee  plan  should  contain  incentives  for  improv¬ 
ing  the  software  process.  Depending  on  the  length  of  the  contract  and  the  initial  maturity 
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of  the  contractor,  the  plan  should  include  intermediate  goals  that  emphasize  the  benefits  of 
advancing  from  a  level  1  maturity  up  to  a  level  5.  Since  it  is  estimated  to  take  at  least  a  year 
to  improve  from  one  level  to  another,  the  award  fee  should  be  staggered.  Assuming  that  the 
contractor’s  initial  maturity  rating  is  a  level  1,  there  should  be  a  minimal  award  at  the  end 
of  the  first  year  for  advancing  from  a  level  1  to  a  level  2.  At  the  end  of  the  second  or  third 
year,  there  should  be  a  larger  reward  for  advancing  from  a  level  2  to  a  level  3,  and  so  on. 
Sample  award  fee  plans  can  be  provided  by  BMDO. 

5.5  Registry  of  Evaluation  Results 

As  described  in  Sections  3, 4,  and  5  the  preparation,  conduct,  and  follow-up  for  an 
SCE  consume  considerable  amounts  of  time,  effort,  and  expense  on  the  part  of  both  the 
Government  and  the  contractor.  Subjecting  a  contractor  to  multiple  SCEs  in  the  context  of 
several  procurements  compounds  this  problem.  Reducing  the  results  of  an  SCE  to  a  mean¬ 
ingful,  concise  form  that  can  be  stored  in  some  repository  and  reused  in  later  procurements 
can  greatly  reduce  this  time,  effort,  and  expense.  Use  of  an  SCE  repository  could  also  short¬ 
en  the  procurement  cycle  considerably  by  reducing  the  number  of  “fresh”  SCEs  that  must 
be  done  in  order  to  evaluate  fairly  and  fully  the  software  development  process  matiuity  of 
all  the  contractors  who  may  respond.  Therefore,  SCE  results  should  be  archived  by  BMDO. 


APPENDIX  A.  RFP  WORDING 


A.l  SCE  Text  For  Inclusion  in  the  RFP 

The  following  sample  text  illustrates  how  SCEs  might  be  inserted  within  Section  L 
or  M  of  the  Request  for  Proposal  (RFP).  This  example  assumes  that  the  SCE  will  be  used 
as  a  specific  criterion  for  source  selection. 

Software  Engineering  Capability.  The  Government  will  evaluate  the  soft¬ 
ware  process  by  reviewing  the  offeror’s  Software  Process  Improvement 
Plan  and  by  using  the  Software  Engineering  Institute  (SEI)  developed  tech¬ 
nique,  Software  Capability  Evaluation  (SCE).  The  Government  will  deter¬ 
mine  the  software  process  capability  by  investigating  the  offeror’s  current 
strengths  and  weaknesses  in  key  process  areas  defined  in  the  SEI  report 
Characterizing  the  Software  Process:  A  Maturity  Framework,  CMU/SET 
87-TR-l  1.  The  Government  will  perform  an  SCE  of  each  offeror  by  review¬ 
ing  current  projects  at  the  site  proposing  on  this  contract  The  evaluation 
will  be  an  organizational  composite,  substantiated  through  individual  inter¬ 
views  and  reviews  of  documentation,  of  the  offeror’s  software  process  prac¬ 
tices  on  selected  projects.  The  evaluation  will  determine  the  offeror’s 
strengths  and  weaknesses  in  key  process  areas  relative  to  maturity  level 
three,  i.e.,  the  extent  to  which  an  offeror  meets  or  exceeds  maturity  level 
three  criteria.  The  on-site  evaluators  may  be  separate  and  distinct  from  the 
proposal  evaluation  team  and  may  include  a  Government  contracting  repre¬ 
sentative.  The  evaluators  will  have  been  trained  in  conducting  SCEs. 

A.2  SCE  Text  for  Inclusion  in  the  IFPP 

The  Instructions  for  Preparation  of  Proposals  (IFPP)  provides  guidance  to  offerors 
as  to  how  they  should  prepare  their  proposal.  The  following  text  requests  the  offeror  to  pro¬ 
vide  Project  Profiles,  organization  charts,  sample  documentation,  and  a  software  process 
improvement  plan.  It  also  requests  the  offeror  to  provide  the  SCE  team  with  facilities  dur¬ 
ing  the  site  visit. 

The  technical  proposal  shall  include  the  offeror’s  response  to  the  software 
evaluation  process.  The  offeror  shall  provide  the  following  information  to 


assist  the  Government’s  preparation  for  the  Software  Capability  Evaluation 
of  each  offeror: 

a.  The  offeror  shall  complete  the  Project  Profile  form  for  six  to  nine  major  soft¬ 
ware  engineering  development  projects. 

All  projects  should  be  drawn  from  the  same  site  and  organization  (e.g.,  profit 
center)  bidding  on  this  solicitation.  One  of  these  projects  must  include  the  [pro¬ 
posed]  software  development  effort  and  the  others  should  be  projects  that  are 
near  completion  or  completed  within  the  last  three  months.  These  projects 
should  be  as  similar  as  possible  in  scope  and  magnitude  to  the  [proposed]  effort. 
The  projects  should  be  from  programs  where  the  offeror  was  the  prime  contrac¬ 
tor,  at  least  one  project  should  include  a  development  where  another  subcon¬ 
tractor  developed  portions  of  the  software,  and  at  least  one  project  should  be  an 
Ada  project,  more  if  applicable.  Project  profiles  from  Special  Access  Programs 
are  discouraged.  For  offerors  with  fewer  than  seven  projects  at  the  bidding  site, 
submit  information  for  as  many  projects  as  are  available. 

Appendix  B  [of  this  report]  contains  an  outline  that  should  be  used  to  generate 
the  project  profiles  for  each  of  the  projects.  Included  in  the  outline  is  the  SEI 
questionnaire.  For  convenience  the  SEI  questions  and  response  form  are  provid¬ 
ed  in  Appendix  C  [of  this  report].  Respond  to  the  SEI  questions  with  a  Yes  or 
No  answer.  For  each  “Yes”  response,  please  note  the  mechanism  or  document 
for  justifying  the  response  on  a  separate  form. 

b.  The  offeror  shall  provide  project-level  and  higher-level  organization  charts. 

The  organization  charts  should  contain  individual’s  names  and  job  titles  and  in¬ 
dicate  how  the  projects  above  are  related  to  each  other.  If  there  are  departments 
that  the  software  projects  rely  on,  these  too  should  be  positioned  on  the  organi¬ 
zation  chart  (e.g.,  training.  Software  Engineering  Process  Group  (SEPG),  qual¬ 
ity  assurance,  configuration  management,  standards,  policy  and  procedures). 

c.  The  offeror  shall  provide  a  sample  Software  Development  Plan  (SDP)  and  a 
Software  Standards  and  Procedure  Manual  (SSPM). 

If  there  are  “generic”  SDPs  and  SSPMs,  those  are  preferred;  otherwise,  select  a 
sample  SDP  and  SSPM  from  the  project  that  has  the  most  representative  SDP. 

d.  The  offeror  shall  submit  its  site’s  Software  Process  Improvement  Plan,  in  the 
format  of  its  choosing,  with  its  proposal. 

The  document  shall  be  no  longer  than  15  pages.  The  Software  Process  Improve¬ 
ment  Plan  shall  be  detailed  enough  for  the  offeror  to  communicate  its  current 
software  process  capability,  specific  planned  improvements,  dedicated  resourc¬ 
es,  effort  estimates,  and  a  time  phasing  of  those  improvements  to  bring  the  off¬ 
eror’s  software  process  maturity  to  the  organization’s  desired  maturity  level. 


After  the  proposal  is  received,  the  government  will  coordinate  a  site  visit  with 
the  offeror  to  discuss  the  questionnaire  responses  and  conduct  the  SCE  at  the 
offeror’s  location. 

The  offeror  shall  provide  a  point  of  contact  and  phone  number  at  the  offeror’s 
site  for  the  SCE  team  leader  to  coordinate  all  SCE  activities.  So  that  the  site  visit 
will  transpire  smoothly  for  both  parties,  the  government  will  also  communicate 
low-level  details  about  the  site  visit  during  the  coordination  process,  e.g.,  inter¬ 
view  schedules,  documentation  requests,  facilities  for  the  evaluation  team.  The 
offeror  shall  be  notified  approximately  two  working  days  prior  to  the  site  visit 
of  the  projects  to  be  examined.  The  site  visit  dates  selected  by  the  government 
are  not  open  for  discussion. 

During  the  site  visit,  the  SCE  team  will  need  a  closed  meeting  room  capable  of 
accommodating  at  least  eight  people. 

The  offeror  shall  have  a  copy  of  the  organization’s  software  standards,  proce¬ 
dures  and/or  operating  instructions,  and  organizational  charts  for  the  projects 
being  reviewed  in  the  meeting  room  when  the  SCE  team  arrives.  All  interviews 
conducted  as  part  of  the  SCE  shall  be  done  in  private,  one  individual  at  a  time. 
The  SCE  team  may  be  separate  and  distinct  from  the  proposal  evaluation  team. 
The  SCE  team  has  been  trained  in  performing  Software  Capability  Evaluations. 

If  security  authorization  is  necessary  for  the  evaluation  team  to  be  on  the  con¬ 
tractor’s  site,  a  FAX  number  and  telephone  number  of  the  contractor’s  security 
office  should  be  provided  along  with  a  list  of  any  other  pertinent  information 
required  to  obtain  security  approval. 

It  is  not  the  intent  of  the  evaluation  to  discuss  classified  information.  It  is  only 
a  matter  of  convenience  for  the  evaluation  team  not  to  require  an  escort  when 
walking  to  the  cafeteria  and  restroom. 


APPENDIX  B.  PROJECT  PROFILE  OUTLINE 


The  following  form  is  a  sample  project  profile  that  can  be  referenced  in  the  Request 
for  Proposal  (RFP).  Six  to  nine  of  these  forms,  for  different  projects,  should  be  filled  out  by 
each  contractor. 

a.  Project  Name;  name  of  project  listed  on  the  contract 

b.  Project  Number;  unique  identifying  number  on  the  contract. 

c.  Project  Type;  e.g.,  scientific,  human-machine,  business,  control,  support  soft¬ 
ware. 

d.  Customer;  the  agency  that  procured  the  software  and  a  point  of  contact  within 
that  organization. 

e.  Subcontractors/Prime  Contractors;  list  any  subcontractors  employed  on  the 
project  or  list  the  prime  contractor  if  the  offeror  was  a  subcontractor. 

f.  Current  Phase;  identify  the  current  phase  of  the  software  development  pro¬ 
cess,  e.g.,  requirements  definition,  detailed  design,  code  and  unit  test,  integra¬ 
tion  test,  maintenance. 

g.  Location;  primary  site  of  the  software  development  effort. 

h.  Start  Date;  starting  date  of  the  contract. 

i.  Design  Completion  Date;  estimated  or  actual. 

j.  Code  Completion  Date;  estimated  or  actual. 

k.  End  Date;  contract  completion  date. 

l.  Team  Size;  peak  staff-month  period  and  average  staff-years  over  the  contract 
period. 

m.  Estimated  KSLOC;  estimated/actual  Thousand  Source  Lines  of  Code. 
(KSLOC) 


n.  Programming  Languages:  percentage  of  KSLOC  in  languages  (e.g.,  Ada,  For¬ 
tran,  Pascal,  C,  Assembly). 

o.  Target  Hardware  System:  computer  on  which  software  executes. 

p.  Development  Hardware  System:  host  computer  for  the  compiler  and  support 
environment. 

q.  Applicable  Standards:  e.g.,  DOD-STD-2167A. 

r.  Cost:  actual/estimated  dollars  spent  to  date/completion. 

s.  SEI  Questionnaire:  The  attached  questionnaire  and  its  answer  sheet  should  be 
completed  for  each  of  the  projects. 

t.  Organization  Chart:  most  recent  organization  chart  for  each  project  with  titles 
and  individual  names.  This  chart  should  identify  the  individual  responsible  for 
the  following  activities:  project  management,  system  engineering,  software 
project  management,  software  engineering,  quality  assurance,  configuration 
management,  subcontractor  control,  simulation,  integration  and  testing,  and 
other  technical  software  activities. 


APPENDIX  C.  SEI  QUESTIONNAIRE  AND  RESPONSE  FORM 


This  form  is  a  modified  version  of  the  Software  Engineering  Institute  (SEI)  Process 
Maturity  Model  (PMM)  questionnaire.  It  should  be  referenced  and  included  in  the  Request 
for  Proposal  (RFP),  and  filled  out  by  each  contractor. 

Name  of  Projects 

Project  A:  _ _ _ 

Project  B:  _ _ _ _ 

Project  C:  _ _ _ 

Project  D: _ 

Project  E: _ _ _ 

Project  F:  - 

Project  G:  _ 

Project  H:  _ _ 


Project  I: 


A 


PROJECT  MANAGEMENT 


2.1.3* 

Is  a  formal  procedure  used  in  the  management 
review  of  each  software  development  prior  to 
making  contractual  commitments? 


2.1.4 

Is  a  formal  procedure  used  to  assure  periodic 
management  review  of  the  status  of  each  software 
development  project? 


2.4.1* 

Does  senior  management  have  a  mechanism  for 
the  regular  review  of  the  status  of  software 
development  projects? 


2.4.7* 

Do  software  development  first-line  managers  sign 
off  on  their  schedules  and  cost  estimates? 


For  each  project  involving  software  development, 
is  there  a  designated  software  manager? 


Does  the  project  software  manager  report  directly 
to  the  project  (or  project  development)  manager? 


Is  a  mechanism  used  for  independently  calling 
integration  and  test  issues  to  the  attention  of  the 
project  manager? 

2.1.17  '  “ 

Is  a  mechanism  used  for  ensuring  that  the 
software  design  teams  understand  each  software 
requirement? 


Is  a  mechanism  used  for  identifying  and  resolving 
system  engineering  issues  that  affect  software? 


Is  software  system  engineering  represented  on 
the  system  design  team? 


2.4.10 

Is  there  a  formal  management  process  for 
determining  if  the  prototyping  of  software  functions 
is  an  appropriate  part  of  design  process? 


C  D  E  F  G  H 


2.4.5 

Is  a  mechanism  used  for  regular  technical 
interchanges  with  the  customer? 


Is  there  a  mechanism ior  assuring  that  software 
subcontractors,  if  any,  follow  a  disciplined 
software  development  process? 


PROJECT  PLANNING 


2.1.14* 

Is  a  formal  procedure  used  to  make  estimates  of 
software  size? 


2.1.16* 

Are  formal  procedures  applied  to  estimating 
software  development  cost? 


2.2.7 

Are  profiles  maintained  of  actual  versus  planned 
software  units  designed,  over  time? 


Are  profiles  maintained  of  actual  versus  planned 
software  units  completing  unit  testing,  over  time? 


2.2.9 

Are  profiles  maintained  of  actual  versus  planned 
software  units  integrated,  over  time? 


2.2.18 

Is  test  progress  tracked  by  deliverable  software 
component  and  compared  to  the  plan? 


2.2.19 

Are  profiles  maintained  of  software  build/release 
content  versus  time? 


2.1.15* 

Is  a  formal  procedure  used  to  product  software 
development  schedules? 


2.2.1* 

Are  software  staffing  profiles  maintained  of  actual 
staffing  versus  planned  staffing? 


2.2.2* 

Are  profiles  of  software  size  maintained  for  each 
software  configuration  item,  over  time? 


A 


B 


2.2.10 

Are  target  computer  memory  utilization  estimates 
and  actuals  tracked? 

2.2.11 

Are  target  computer  throughput  utilization 
estimates  and  actuals  tracked? 

2.2.12 

Is  target  computer  I/O  channel  utilization  tracked? 
QUALITY  ASSURANCE 
1.1.3* 

Does  the  QA  function  have  a  management 
reporting  channel  separate  from  the  software 
development  project  management? 

2.1.7 

For  each  project,  are  independent  audits 
conducted  for  each  step  of  the  software 
development  process? 

2.4.19* 

Is  a  mechanism  used  for  verifying  that  samples 
examined  by  QA  are  truly  representative  of  the 
work  performed? 

2.4.6* 

Is  a  mechanism  used  for  ensuring  compliance 
with  the  software  engineering  standards? 

CONFIGURATION  MANAGEMENT 


1.1.6* 

Is  there  a  software  configuration  control  function 
for  each  project  that  involves  software 
development? 

2.4.9* 

Is  a  mechanism  used  for  controlling  changes  to 
software  requirements? 

2.4.17* 

Is  a  mechanism  used  for  controlling  changes  to 
the  code?  (Who  can  make  changes  and  under 
what  circumstances?) 

2.4.13* 

Is  a  mechanism  used  for  controlling  changes  to 
the  software  design? 


A 

B 

C 

D 

1.1.4 

Is  there  a  designated  individual  or  team 
responsible  for  the  control  of  software  interfaces? 


Is  a  mechanism  used  for  ensuring  traceability 
between  the  software  requirements  and  top-level 
design? 


2.4.11 

Is  a  mechanism  used  for  ensuring  traceability 
between  the  software  top-level  and  detailed 
designs? 


2.4.14 

Is  a  mechanism  used  for  ensuring  traceability 
between  the  software  detailed  design  and  the 
code? 


2.4.18 

Is  a  mechanism  used  for  configuration 
management  of  the  software  tools  used  in  the 
development  process? 


2.4.20 

Is  there  a  mechanismior  assuring  that  regression 
testing  is  routinely  performed? 


2.4.21* 

Is  there  a  mechanismior  assuring  the  adequacy 
of  regression  testing? 


PEER  REVIEWS 


2.4.12* 

Are  internal  software  design  reviews  conducted? 


2.4.16* 

Are  software  code  reviews  conducted? 


2.4.22 

Are  formal  test  case  reviews  conducted? 


2.2.13* 

Are  design  and  code  review  coverages  measured 
and  recorded? 


2.2.4* 

Are  statistics  on  software  code  and  test  errors 
gathered? 


2.2.3* 

Are  statistics  on  software  design  errors  gathered? 


2.3.2* 

Are  the  review  data  gathered  during  design 
reviews  analyzed? 


2.3.8* 

Is  review  efficiency  ana\yze6  for  each  project? 


2.2.16 

Are  software  trouble  reports  resulting  from  testing 
traced  to  closure? 


2.2.15* 

Are  the  action  items  resulting  from  design  reviews 
tracked  to  closure? 


2.2.17* 

Are  the  action  items  resulting  from  code  reviews 
tracked  to  closure? 


TESTING 


2.2.14* 

Is  test  coverage  measured  and  recorded  for  each 
phase  of  functional  testing? 


TRAINING 


Is  there  a  required  training  program  for  all  newly 
appointed  development  managers  designed  to 
familiarize  them  with  the  software  project 
management? 


Is  there  a  required  software  engineering  training 
program  for  first-line  supervisors  of  software 
development? 


1.2.5* 

Is  a  formal  training  program  required  for  design 
and  code  review  leaders'? 


1.2.3* 

Is  there  a  required  software  engineering  training 
program  for  software  developers? 


A 

B 

C 

D 

E 

F 

G 

H 

STANDARDS  AND  PROCEDURES 


2.1.9 

Are  coding  standards  applied  to  each  software 
development  project? 


2.1.6 

Are  standards  applied  to  each  software 
development  project? 


2.1.11 

Are  code  maintainability  standards  applied? 


2.1.10 

Are  standards  applied  to  the  preparation  of  unit 
test  cases? 


2.1.18 

Are  man-machine  interface  standards  applied  to 
each  appropriate  software  development  project? 


2.1.12 

Are  internal  design  review  standards  applied? 


2.1.13* 

Are  code  review  standards  applied? 


SOFTWARE  ENGINEERING  PROCESS 
GROUP 


1.1.7* 

Is  there  a  software  engineering  process  group 
function? 


2.1.1* 

Does  the  software  organization  use  a 
standardized  and  documented  software 
development  process  on  each  project? 


2.1.2 

Does  the  standard  software  development  process 
documentation  describe  the  use  of  tools  and 
techniques? 


2.4.15 

Are  formal  records  maintained  of  unit  (module) 
development  progress? 


2.3.1* 

Has  a  managed  and  controlled  process  database 
been  established  for  process  metrics  data  across 
all  projects? 

2.3.9 

Is  the  software  productivity  analyzed  for  major 
process  steps? 

2.2.5* 

Are  design  errors  projected  and  compared  to 
actuals? 

2.2.6* 

Are  code  and  test  errors  projected  and  compared 
to  actuals? 

2.3.3* 

Is  the  error  data  from  code  reviews  and  tests 
analyzed  to  determine  the  likely  distribution  and 
characteristics  of  the  errors  remaining  in  the 
product? 

KEY  PROCESS  AREAS 
2.4.2* 

Is  a  mechanism  used  for  periodically  assessing 
the  software  engineering  process  and 
implementing  indicated  improvements? 

2.3.4* 

Are  analyses  of  errors  conducted  to  determine 
their  process  related  causes? 

2.3.5* 

Is  a  mechanism  used  for  error  cause  analysis? 
2.3.6* 

Are  the  error  causes  reviewed  to  determine  the 
process  changes  required  to  prevent  them? 

2.3.7* 

Is  a  mechanism  used  for  initiating  error  prevention 
actions? 


APPENDIX  D.  CHECKLIST 


Project  Management 


Software  managers  play  an  active  roll  in  pro¬ 
posal  preparation. 

Software  progress  is  tracked.  Managers  fre¬ 
quently  review  the  progress. 

Software  schedules  and  cost  estimates  are 
approved  by  the  software  managers. 

There  is  a  process  for  raising,  tracking,  and 
resolving  issues. 

Software  managers  are  in  the  reporting  chain. 
Software  requirements  team  relate  to  software 
design  team. 

Managers  have  visibility  into  integration  &  test. 
System  engineering  &  software  engineering 
relationship  is  defined  and  trade-offs  are  made. 


•  Subcontractor’s  development  process  is  known 
and  monitored. 

•  Subcontractor’s  standards,  procedures,  process 
comply  with  prime’s. 

•  Subcontractor’s  results,  performance,  and  com¬ 
mitments  are  tracked. 

•  Prime’s  subcontract  manager  is  knowledgeable 
of  software  &  trained. 

•  Subcontractors  products  undergo  periodic  tech¬ 
nical  reviews  &  interchange  with  prime. 

•  Prime’s  QA  &  CM  monitors  subcontractor’s 
QA  &  CM. 

•  Prime’s  senior  management  reviews  status  of 
subcontractors  regularly. 


Project  Planning 


Metrics:  design  progress,  test  progress,  staffing, 
integration  progress  are  used. 

Metrics:  software  size  over  time,  memory  utili¬ 
zation,  throughput,  I/O  channel  utilization  are 
used. 

Development  progress  is  tracked  and  reported 
to  program  management  regularly. 

Software  size,  cost,  &  schedules  are  estimated 
according  to  documented  procedures 
Commitments  are  obtained  and  documented  for 
size,  cost,  schedules,  and  resources. 


Policy  exists  for  resource  planning  &  commit¬ 
ments. 

Software  managers  are  trained  on  software  esti¬ 
mation  procedures. 

Actual  vs.  planned  estimates  are  recorded  and 
compared. 

Central  estimation  manager  &  data  base  are 
maintained  for  tracking  and  improving  accura¬ 
cy. 


Quality  Assurance  (QA) 


Independent  QA  reporting  chain  exists. 

Audits  are  conducted  during  all  phases  of  soft¬ 
ware  life  cycle  &  on  all  line  activities. 

Audits  are  representative. 

QA  has  adequate  resources  (3%). 


QA  audits  subcontractors. 

Deviations  are  handled  according  to  document¬ 
ed  procedures. 

Senior  management  reviews  QA  activities  reg¬ 
ularly. 

QA  authority  &  concurrence  are  required. 


Configuration  Management  (CM) 


CM  controls  requirements,  design,  code,  and 
interface  changes. 

Traceability  is  maintained  from  requirements  to 
design  to  code. 

Tool  is  used  to  help  control  versions  and  buDds. 
Regression  testing  is  performed. 

Baselines  are  established  and  include  tools  and 
change  log  in  addition  to  requirements,  design, 
code,  and  tests. 

CM  plan  includes  staff,  schedule,  responsibili¬ 
ties,  resources,  tools,  &  facilities. 


•  Library  system  stores  work  products  and  pre¬ 
vents  unauthorized  changes. 

•  Documented  change  request  process  includes 
check  in/out,  reviews,  regression  tests,  and 
baseline  descriptions. 

•  Change  Control  Board  &  change  proposal  pro¬ 
cess  are  documented. 

•  Change  log  is  used  to  track  open/closed  change 
requests. 


Peer  Reviews 


Design,  code,  test  case  peer  reviews  are  con¬ 
ducted. 

Attendees  include  no  management,  interface 
leads,  peers,  several  people  (more  than  one). 
Documented  procedures  &  checklists  are  used, 
Peer  review  process  is  in  SDR 


•  Peer  review  schedule  and  list  of  attendees  are 
published. 

•  Statistics  are  collected  identifying  type  of 
errors,  severity,  location,  time  preparing, 
reviewing,  and  correcting. 

•  Errors  are  tracked  to  closure. 

•  QA  audits  peer  review  activities. 


Training 


•  CM  and  QA  leaders  are  trained. 

•  Project  managers  are  trained  on  software  esti¬ 
mation  and  peer  reviews. 

•  Software  supervisors  are  trained  on  QA,  CM, 
software  estimation,  and  peer  reviews. 

•  Software  developers  are  trained  on  peer 
reviews,  software  development  process,  and 
tools. 


•  Training  resources  include  money,  facilities, 
tools,  and  schedules. 

•  Training  policy  exists. 

•  Projects  have  training  needs  identified. 

•  Job  functions  are  mapped  to  training. 

•  Training  records  are  maintained  and  include 
people  and  courses  attended. 


Standards  and  Procedures 

•  Standards  are  enforced. 

•  Standards  are  updated  as  needed. 

•  Responsibility  for  updating  standards  &  policy 
is  assigned. 

Software  Engineering  Process  Group  (SEPG) 


•  Coding,  unit  development  folders,  and  man- 
machine  interface  standards  exist. 

•  Generic  Software  Development  Plan,  QA  plan, 
CM  plan,  and  standards  exist. 


Organization  is  focused  on  standard  software 
process  &  improvement. 

Strengths  &  weaknesses,  are  assessed,  plans  to 
improve  are  developed. 

Activities  include  reuse,  computer-assisted 
software  engineering,  measurement,  training, 
process  definition,  and  improvement. 

SEPG  plan  includes  activities,  schedules, 
responsibility,  and  resources. 


SEPG  is  informed  of  projects  process:  new 
techniques,  technology  &  problems. 

Repository  of  software  process  information  is 
maintained  (life  cycle  models,  lessons). 
Projects  have  received  benefits  of  SEPG. 
Projects  provide  input  to  SEPG  activities. 

SEPG  has  mechanism  to  transfer  technology  to 
projects. 

Full-time  SEPG  resources  are  available. 


APPENDIX  E.  SAMPLE  FINDINGS 


1.  PROJECT  MANAGEMENT 

1.1  Sample  Strengths 

a.  Software  leads  have  direct  reporting  to  Project  Management. 

b.  Project  management  and  software  leads  are  regularly  informed  of  software 
development  status  and  issues. 

c.  Compliance  with  project  plans  is  monitored  and  enforced  (e.g..  Quality  Assur¬ 
ance  (QA)). 

d.  Management  has  visibility  into  software  issues.  Issues  are  elevated  and  tracking 
process  in  place. 

e.  Subcontract  management  process  is  well  defined  and  enforced. 

f.  Software  engineering  is  represented  throughout  system  engineering  process 
(e.g.,  concurrent  engineering  teams). 

g.  Software  estimates  and  commitments  are  reviewed  and  approved  by  senior 
management 

h.  Project  responsibilities  for  each  life  cycle  phase  are  documented. 

i.  Regular  technical  interchanges  are  held  with  the  customer. 

1.2  Sample  Weaknesses 

a.  Lines  of  authority  to  support  organizations  are  not  clearly  understood  (by  peo¬ 
ple  in  the  organization). 

b.  Project  management  procedures  remain  informal  and  non-standard  across 
projects  (i.e.,  not  formally  or  consistently  documented).  Examples  are  risk  man¬ 
agement  procedures,  procedure  for  escalation  and  tracking  of  issues  to  senior 
management,  project  plans,  and  the  concurrent  engineering  team  concept. 


E-1 


2.  PROJECT  PLANNING 


2.1  Sample  Strengths 

a.  Process  for  cost  estimation  is  well  defined  and  is  based  in  reality  (i.e.,  cost  esti¬ 
mation  data  is  maintained  in  a  database). 

b.  Corporate  historical  databases  are  available  for  the  sizing  tools  and  updated 
periodically  with  cost,  sizing,  and  quality  metrics  collected  from  all  projects. 

c.  A  central  group  is  responsible  for  maintaining  a  central  cost-and-size  database. 

d.  Guidelines  exist  for  developing  software  size,  cost,  and  schedule  estimates. 

e.  High-level  software  schedules  and  costs  are  monitored  regularly  (e.g.,  mile¬ 
stone  reports). 

f.  Project-level  software  metrics  data  are  being  collected  (progress,  utilization, 
staffing,  errors,  earned  value). 

2.2  Sample  Weaknesses 

a.  No  formal  process  is  in  place  for  estimating  software  resources  (lines  of  code, 
effort). 

b.  Software  managers  are  not  trained  in  software  size  estimation  methods. 

c.  Software  schedules  are  not  derived  from  a  defined  and  repeatable  process. 

d.  No  central  software  estimation  personnel,  data  base,  or  feedback  mechanism 
exists. 

e.  Software  development  progress  monitoring  is  inconsistent  for  computer  soft¬ 
ware  configuration  items,  computer  software  components,  and  computer  soft¬ 
ware  units. 

f.  No  standardized  software  measurements  exist  to  track  progress  and  product 
quality  consistently  across  projects. 

g.  Metrics  (e.g.,  size,  cost,  schedules)  are  being  collected  at  the  project  level  but 
not  in  a  consistent  or  effective  manner. 

h.  Hardware  utilization  decomposition  and  tracking  (memory,  throughput)  is  lack¬ 
ing. 

i.  No  enforceable  organization-wide  directive  exists  for  use  of  estimating  tools. 
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3.  QUALITY  ASSURANCE 


3.1  Sample  Strengths 

a.  Authority  and  responsibilities  are  well  defined  and  documented  for  each  phase 
of  life  cycle  and  activities. 

a.  QA  has  separate  reporting  chain  to  senior  management. 

b.  QA  concurrence  is  required  for  major  transitions  in  development. 

c.  QA  has  adequate  resources. 

d.  QA  has  a  visible  audit  trail  through  all  phases  of  software  development  and  all 
line  activities  (e.g.,  configuration  management  (CM)). 

e.  Well-documented  policies  and  procedures  are  consistent  across  programs  (e.g., 
audits,  checklists). 

3.2  Sample  Weaknesses 

a.  Combining  the  test  function  with  the  QA  function  may  negate  the  advantage  of 
a  separate  reporting  path  to  senior  management. 

b.  No  mechanism  exists  to  determine  if  audits  are  representative  and  sufficient. 

c.  QA  personnel  are  also  performing  some  line  activities  (e.g.,  CM,  testing). 

d.  No  feedback  mechanism  exists  for  improving  software  quality  process. 

4.  CONFIGURATION  MANAGEMENT 
4.1  Sample  Strengths 

a.  Configuration  control  board  responsibilities  are  defined  and  implemented. 

b.  Project  change  control  processes  are  defined  and  documented. 

c.  Automated  tools  are  used  to  enforce  change  control  process  on  projects. 

d.  Configuration  status  account  report  is  produced. 

e.  Strong  traceability  is  shown  between  development  products. 

f.  Formal  mechanism  is  in  place  for  tracking  problem  reports. 

g.  Feedback  mechanism  is  in  place  to  refine  division-wide  CM  practice. 


4.2  Sample  Weaknesses 

a.  Standard  software  configuration  control  is  lacking  across  all  projects. 

b.  Regression  testing  is  not  performed  and  reported. 

c.  No  CM  training  is  performed  at  the  project  level. 

a.  Traceability  between  requirements,  design,  and  code  is  inadequate. 

b.  No  organization-wide  policies  and  procedures  exist  for  developmental  CM 
practices. 

c.  No  feedback  mechanism  exists  to  track  and  refine  CM  practices. 

5.  TRAINING 

5.1  Sample  Strengths 

a.  Comprehensive  training  requirements  for  career  progression  exist  for  manage¬ 
ment  and  technical  personnel. 

b.  Senior  management  is  committed  to  training. 

c.  Senior  management  is  responsive  to  training  needs. 

d.  Comprehensive  training  manuals  and  guidebooks  are  provided. 

e.  Course  schedules  are  documented. 

5.2  Sample  Weaknesses 

a.  Organizational  training  requirements  for  software  personnel  are  lacking  (e.g., 
introductory,  managers,  task  specific). 

b.  Training  records  are  not  maintained  (e.g.,  completed  course  work  per  person). 

c.  Routine  training  schedules  are  not  identified. 

d.  No  training  is  provided  in  support  functional  areas  (e.g.,  CM,  QA). 

e.  No  formal  feedback  mechanism  exists  to  determine  adequacy  of  training  (cri¬ 
tique  forms). 

f.  A  process  for  identification  of  project  training  requirements  is  lacking. 

g.  Centralized  administration  of  a  group  training  program  is  lacking. 


6.  STANDARDS  AND  PROCEDURES 


6.1  Sample  Strengths 

a.  Responsibility  has  been  assigned  for  standards  development  and  maintenance. 

b.  Audit  criteria  has  been  established  and  audits  are  routinely  performed. 

c.  Management  is  committed  as  evidenced  by  reviews  of  compliance. 

d.  Well-defined  corporate/group  and  division/company  level  standards  and  proce¬ 
dures  exist. 

e.  A  Software  Development  Plan  (SDP)  template  is  developed  to  facilitate  consis¬ 
tency  among  project  SDPs. 

f.  Review  and  sign-off  on  all  project  SDPs  facilitates  the  flowing  of  the  process 
guidance  to  the  projects. 

g.  Deviations  from  group  and  division  policies  require  policy  control  board 
approval. 

6.2  Sample  Weaknesses 

a.  No  review  process  exists  to  ensure  policy,  standards,  and  procedures  are  current 
and  effective. 

b.  Standards  and  practices  for  software  engineering  are  not  enforced  at  the  project 
level. 

c.  Process  for  developing  project  standards  is  ad  hoc. 

d.  No  formal  process  exists  for  tailoring  software  standards  and  procedures. 

7.  SOFTWARE  ENGINEERING  PROCESS  GROUP  (SEPG) 

7.1  Sample  Strengths 

a.  Group-level  SEPG  is  active  with  full  time  staff. 

b.  SEPG  is  focused  on  continuous  process  improvement. 

c.  Periodic  self-assessments  are  performed  by  trained  personnel. 

d.  Process  improvement  action  plan  is  in  place. 

e.  Process  repository  is  established  (e.g.,  tools,  size,  cost  data). 


f.  SEPG  facilitates  flow  of  information  between  divisions. 


7.2  Sample  Weaknesses 

a.  No  feedback  mechanism  exists  for  SEPG  users — SEPG  activities  are  not  effec¬ 
tively  communicated  to  all  project  software  development  personnel. 

b.  No  process  metrics  exist  to  measure  effectiveness  of  processes. 

c.  No  technology  insertion  plan  exists. 

8.  WALKTHROUGHS 

8.1  Sample  Strengths 

a.  Policy  and  procedures  exist  to  support  the  walkthrough  process  at  all  phases  of 
the  software  life  cycle  (requirements,  design,  code). 

b.  Guidelines  define  walkthrough  process  activities  (e.g.,  attendance,  preparation, 
review,  checklists,  reporting  results). 

c.  Formal  reporting  and  tracking  of  walkthrough  findings  are  performed  at  project 
level. 

d.  QA  verifies  closure  of  problems  raised  during  walkthroughs. 

e.  Walkthrough  process  is  applied  to  subcontractors. 

f.  Announcements  of  walkthroughs  are  made  and  contents  are  distributed  in 
advance. 

8.2  Sample  Weaknesses 

a.  Execution  of  walkthrough  guidelines  across  projects  is  inconsistent  (i.e.,  no 
code  and  test  case  walkthroughs). 

b.  No  walkthrough  summary  data  and  analysis  exist  to  determine  effectiveness  of 
walkthrough  process  (e.g.,  is  sufficient  time  spent  preparing)  and  to  improve  the 
development  process  (i.e.,  identify  reoccurring  problems). 

c.  Level  and  composition  of  participants  is  of  concern  (e.g.,  management  attends, 
audience  can  be  as  large  as  20). 

d.  No  training  courses  are  available  for  walkthrough  participants  (e.g.,  mediator, 
reviewer,  developer). 


E-6 


REFERENCES 


[AFSC  1991] 

Air  Force  Systems  Command.  1992.  Pamphlet  800-5,  Acquisition 
Management:  Software  Development  Capability  Capacity  Review. 
Wright-Patterson  AFB,  OH. 

[DoD  1987] 

Department  of  Defense.  1987.  DOD-STD-2167A,  Defense  Software 
Development  Standard. 

[FAR  1992] 

Federal  Acquisition  Regulations.  February  1992.  Chicago,  IL: 
Commerce  Clearing  House. 

[Humphrey  1987] 

Humphrey,  W.S.,  and  W.L.  Sweet.  1987.  A  Method  for  Assessing 
the  Software  Engineering  Capability  of  Contractors.  SEI  Technical 
Report  SEI-87-TR-23.  Pittsburgh,  PA:  Software  Engineering  Insti¬ 
tute. 

[Humphrey  1989] 

Humphrey,  W.S.  1989.  Managing  the  Software  Process.  Reading, 
MA:  Addison- Wesley. 

[SEI  1991a] 

Software  Engineering  Institute.  1992.  Evaluation  Team  Training: 
Participant’s  Handbook.  Pittsburgh,  PA:  SEI. 

[SEI  1991b] 

Software  Engineering  Institute.  1992.  Capability  Maturity  Model 
for  Software.  Pittsburgh,  PA:  SEI. 

[SPR  1991] 

Software  Productivity  Research.  February  1991.  Checkpoint  Ques¬ 
tionnaire,  Version  3.0.  Burlington,  MA:  SPR. 

[Springsteen  1991] 

Springsteen,  B.  and  Dennis  Fife.  1991.  Software  Maturity  Model 
Applied  to  SDI.  IDA  Document  D-I042.  Alexandria,  VA:  Institute 
for  Defense  Analyses. 

References- 1 


LIST  OF  ACRONYMS 


BE 

Brilliant  Eyes 

BMD 

Ballistic  Missile  Defense 

BMDO 

Ballistic  Missile  Defense  Organization 

BP 

Brilliant  Pebbles 

CM 

Configuration  Management 

CMM 

Capability  Maturity  Model 

Denx/Val 

DemonstrationA^alidation 

DoD 

Department  of  Defense 

EMD 

Engineering  Manufacturing  and  Development 

FAR 

Federal  Acquisition  Regulation 

FFRDC 

Federally  Funded  Research  and  Development  Center 

GBI 

Ground  Based  Interceptor 

GBR 

Ground  Based  Radar 

IDA 

Institute  for  Defense  Analyses 

IFPP 

Instructions  for  Preparation  of  Proposals 

KPA 

Key  Process  Area 

KSLOC 

Thousand  Source  Lines  of  Code 

MIS 

Management  Information  System 

MMI 

Man-Machine  Interface 

NTBIC 

National  Test  Bed  and  Integration  Contractor 

NTF 

National  Test  Facility 

PIC 

Procurement  Integrity  Certificate 

PMM 

Process  Maturity  Model 

PO 

Program  Office 

QA 

Quality  Assurance 

RFP 

Request  for  Proposal 

SCE 

Software  Capability  Evaluation 

SDC/CR 

Software  Development  Capability/Capacity  Review 

SDP 

Software  Development  Plan 

Acronyms- 1 


SEI 

Software  Engineering  Institute 

SEPG 

Software  Engineering  Process  Group 

SPA 

Software  Process  Assessment 

SPR 

Software  Productivity  Research 

SSA 

Source  Selection  Authority 

SSPM 

Software  Standards  and  Procedures  Manual 

SSEB 

Source  Selection  Evaluation  Board 

TBD 

To  Be  Determined 

TSDM 

Trusted  Software  Development  Methodology 

Acronyms-2 


REPORT  DOCUMENTATION  PAGE 


Form  Approved 
0MB  No.  0704-0188 


ftiblic  rq5orting  burden  for  thus  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existmg  data  sources, 
gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this 
collection  of  information,  including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson 
Davis  Highway.  Suite  1204,  Arlington,  VA  22202-4302,  and  to  the  Office  of  Management  and  Budget,  Paperwork  Reduction  Project  (0704-0188),  Washington,  DC  20503. 


1.  AGENCY  USE  ONLY  (Leave  blank) 


2.  REPORT  DATE 

October  1994 


3.  REPORT  TYPE  AND  DATES  COVERED 

Final 


4.  TtriE  AND  SUBTITLE 

Conducting  Software  Capability  Evaluations 


5.  FUNDING  NUMBERS 

DASW01-94-C-0054 


6.  AUTHOR(S) 

Dennis  W.  Fife,  BiU  Brykezynski,  Deborah  Heystek,  Robert  J. 
Knapper,  Beth  Springsteen 


Task  Order  T-R2-597.2 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Institute  for  Defense  Analyses  (IDA) 

1801  N.  Beauregard  St. 

Alexandria,  VA  223 1 1-1772 


8.  PERFORMING  ORGANIZATION  REPORT 
NUMBER 


IDA  Paper  P-2771 


9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

BMDO/DB 

Room  1E149,  The  Pentagon 
Washington,  D.C.  20301-7100 


10.  SPONSORING/MONITORING  AGENCY 
REPORT  NUMBER 


12a.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release,  unlimited  distribution:  January  31, 
1995. 


12b.  DISTRIBUTION  CODE 


13.  ABSTRACT  (Maximum  200  words) 

This  document  presents  the  results  of  an  ongoing  study  to  assess  and  recommend  procedures  for 
conducting  a  Software  Capability  Evaluation  (SCE)  for  Ballistic  Missile  Defense  Organization 
(BMDO)  programs.  BMDO  is  using  the  SCE  methodology  of  the  Software  Engineering  Institute 
(SEI)  to  evaluate  contractors  during  the  source  selection  process  and  to  monitor  contractors  after  the 
award  has  been  granted.  SEI  provides  basic  procedures  in  its  SCE  training  course,  but  during  the 
course  of  conducting  more  than  seven  SCEs,  IDA  found  that  additional  procedures  and  information 
were  necessary  to  improve  the  effectiveness  of  the  BMD  SCE  teams.  This  document  provides 
additional  information  in  the  form  of  lessons  learned,  findings,  and  recommended  procedures  and 
techniques  for  conducting  SCEs  on  BMD  contractors.  This  paper  will  be  used  to  supplement  SEI’s 
training  material  and  reports. 


14.  SUBIECT  TERMS 

Software,  Evaluations,  Program  Management,  Software  Selection  Criteria, 
Models,  Quality  Assurance,  Configuration  Management,  Standards. 


15.  NUMBER  OF  PAGES 
82 


16.  PRICE  CODE 


17.  SECURITY  CLASSIFICATION  18.  SECURITY  CLASSIFICATION  19.  SECURITY  CLASSIFICATION 
OF  REPORT  OF  THIS  PAGE  OF  ABSTRACT 


Unclassified 


NSN  7540-01-280-5500 


Unclassified 


Unclassified 


20.  LIMITATION  OF  ABSTRACT 


Standard  Form  298  (Rev.  2-89) 
Prescribed  by  ANSI  Std  Z39-18 
298-102 


